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Abstract. We investigate in the paper general (not necessarily definite) canonical systems of difi'erential 
equation in the framework of extension theory of symmetric linear relations. For this aim we first 
introduce the new notion of a boundary relation F : f)^ — > Ho ffi Hi for A*, where f) is a Hilbert space, 
A is a symmetric linear relation in i5,'Ho is a boundary Hilbert space and Hi is a subspace in Ho- 
Unlike known concept of a boundary relation (boundary triplet) for A* our definition of F is applicable 
to relations A with possibly unequal deficiency indices n±{A). Next we develop the known results on 
minimal and maximal relations induced by the general canonical system Jy'{t) — B{t)y{t) = A{t)f{t) on 
an interval X = (a, b), — oo < a < b < oo and then by using a special (so called decomposing) boundary 
relation for T^j^ax we describe in terms of boundary conditions proper extensions of T^in in the case of the 
regular endpoint a and arbitrary (possibly unequal) deficiency indices n±{T^in) ■ If the system is definite, 
then decomposing boundary relation F turns into the decomposing boundary triplet 11 = \H,To,^i} 
for Tmax- Using such a triplet we show that self-adjoint decomposing boundary conditions exist only for 
Hamiltonian systems; moreover, we describe all such conditions in the compact form. These results are 
generalizations of the known results by Rofe-Beketov on regular differential operators. We characterize 
also all maximal dissipative and accumulative separated boundary conditions, which exist for arbitrary 
(not necessarily Hamiltonian) definite canonical systems. 



1. Introduction 

Assume that is a Hilbert space, A is a closed symmetric linear relation in and A* is the adjoint 
linear relation of A. Moreover, denote by [i3i,i02] the set of all bounded operators between S)i and Jo2 
and let [S)] — 

Recall [13, 23] that a triplet 11 = {"H, Fq, Fi}, where % is an auxihary Hilbert space and Fo, Fi -.A* — > "H 
are (boundary) linear maps, is called a boundary triplet for A* if the map F := (Fo Fi)^ -.A* — > H "H 
is surjective and the following "abstract Green's identity" holds 

(1.1) (/',5)-(/,.9') = (ri/,rog)-(Fo/,Fi5), / = {/,/'}, g^{g,g'}^A\ 

In [7, 23] an abstract Weyl function Mn(A) was associated with a boundary triplet H. This function 
is defined for all A e C \ M by the equality 

(1.2) Fi{/a, A/a} = Afn(A)Fo{/A, A/a}, fx G ker {A* - A). 

It turns out that M(A) is a Nevanlinna [H]-valued function, i.e., M{X) is holomorphic on C\R, M*(A) = 
Af (A) and ImA • ImM(A) > 0, A € C \ M. Moreover, the Nevanlinna function A/(A) is uniformly strict, 
that is e p(ImAf(A)), A e C \ M. 

By choosing a suitable boundary triplet for a concrete problem one can parametrize various classes 
of extensions A D A in the most convenient form. Moreover, the Weyl function enables to characterize 
spectra of extensions A in the similar way as classical m-functions in the spectral theory of Sturm-Liouville 
operators and Jacobi matrices. These and other reasons made a boundary triplet and the corresponding 
Weyl function the convenient tools in the extension theory of symmetric operators (linear relations) and 
its applications (see [13, 7, 23] and references therein). At the same time the theory of boundary triplets 
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and their Weyl functions was developed in [13, 7, 23] only for symmetric relations A with equal deficiency 
indices n+{A) = n-{A). 

To cover the case n+{A) ^ n-{A) we generalized in [25] definition of a boundary triplet as follows. 
Assume that Ho is a Hilbert space, Hi is a subspace in Ho and Tj : A* — >• Hj, j G {0, 1} are linear maps. 
Then a collection If = {Ho ® Hi, Fq, Fi} is a boundary triplet (a D-triplet in terminology of [25]) for A* 

if the map F :— (Fg Fi)^ : A* Ho ® Hi is surjcctive and the identity 

(1.3) (/',5)-(/,5') = (ri/,Fo5)-(Fo/,Fig) + z(P2Fo/,P2Fo.g), ./ = {/,/'}, g = {9,9'}eA* 

holds in place of (1.1) (here P2 is the orthoprojector in Ho onto H2 ■= Ho Q Hi). Associated with such 
a triplet 11 is the Weyl function Mn+(A) defined for all A e C+ by 

(1-4) ri{/A, a/a} = Mn+(A)Fo{/A, A/a}, fx e ker [A* - A) 

The function Mn+(A) is holomorphic on C+, takes on values in [Ho, Hi] and possesses a number of 
properties similar to those of the Weyl function (1.2). In particular, the function Mn(A) = Mn+(A) \ Hi 
is a uniformly strict Nevanlinna function with values in [Hi]. 

A boundary triplet {Ho © 'Hi, Fq, Fi} for A* enables to parametrize efficiently all proper extensions of 
A. Namely, if /C is a Hilbert space and {(Co, Ci); /C} is a pair of operators Cj G [Hj,K,], then the equality 
(the abstract boundary condition) 

(1.5) A = {fGA*: CoTof + CiTif = 0} 

defines the proper extension A C A C A* and conversely each such an extension A admits a unique 
representation (1.5). Moreover, the extension A is maximal dissipative, maximal accumulative or self- 
adjoint if and only if the operator pair {(Co,Ci);/C} belongs to one of the special classes introduced in 
[24]. 

It turns out that each boundary triplet 11 = {Ho ® "Hi, Fq, Fi} satisfies the relation 

(1.6) dim^i = n-{A) < n+{A) = dim Ho 

and, therefore, it is applicable to symmetric relations A with unequal deficiency indices. Clearly, in the 
case Ho = Hi =: H such a triplet 11 and the corresponding Weyl function Mn(A) turn into the similar 
objects in the sense of [13, 23]. 

In [8] the notion of a boundary triplet 11 = {■H,Fo,Fi} for A* has been extended to the case where 
the corresponding Weyl function Afn(A) is a (not necessarily imiformly strict) Nevanlinna function such 
that ^ ap{lmM{i)). Next, the concepts of a boundary relation and its Weyl family which generalize 
the above notions of a boundary triplet and its Weyl function were introduced in [5]. According to [5] a 
boundary relation for A* is a (possibly multivalued) linear map r (Fq Fi)"^ -.Sf^HOiH such that 
domF is dense in A* , the Green's identity (1.1) holds and a certain maximality condition is satisfied. 
The Weyl function of the boundary relation F is defined by 

M(A) = {{ro{/A, A/A},ri{/A,A/A}} : fx e ker (A* - A)}, A e C\M 

and now it belongs to the class of Nevanlinna families; moreover, if the map Fo is surjective, then M(A) 
is a Nevanlinna operator function. In the paper [6] the Weyl function was used for description of various 
classes of the exit space extensions A{= A*) D A. 

In the present paper the new concept of a boundary relation for A* with possibly unequal boundary 
spaces Ho and Hi is introduced. Roughly speaking this relation is a (possibly multivalued) linear map 
r := (Fq Fi)^ : Ho®Hi such that domF = A*, the Green's identity (1.3) holds and a certain 

maximality condition is satisfied (here as before Ho is a Hilbert space and 'Hi is a subspace in 'Ho). 
Moreover, by means of the equality 



(1.7) 



M+(A) = {{FoI/a, A/A},ri{/A,A/A}} : fx & ker (A* - A)}, A e C+ 
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we associate with a boundary relation F the Weyl family M+(A). 

In the paper we study substantially the boundary relations F : i^^ ^ Ho © Hi with dim "Ho < oo. We 
show that in this case domF = A* and there is a boundary triplet Up = {/Co ® /Ci, Go, Gi} for A* with 
ICj e Hj, j e {0, 1} such that F can be represented roughly speaking as a direct sum of (the graph of) 
the operator G = {Gq Gi)'^ and mulF. 

The multivalued part mulF which is a linear relation from Hq to Hi is of importance in our consider- 
ations. If mulF is the operator, then the corresponding Weyl family Af_|-(A) is the holomorphic operator 
function with values in [Ho,Hi], which admits the block representation by means of the Weyl function 
Mur+W of the boundary triplet Ilr and mulF. In the case Ho = Hi =: H one has also ICq = Ki =: K, 
and the mentioned representation of M{\) is 

(1.8) M(A) = p)j:lC®lC^ ^IC® /C^, 

where F and F' are the operators defined in terms of mulF. The equality (1.8) shows that M(A) is a 

Nevanlinna function and Aduri^) is the uniformly strict part of M(A). 

Note that for the boundary relation F : i^^ — > Hq ® Hi with AvaiHo < oo the equalities 

dim 7^0 = n'+{A) + dim(mulF), dim Hi = n-{A) + dim(mulF) 

are valid (c.f. (1.6)). so that n-{A) < n-i-(^) < oo. At the same time in the case of unequal deficiency 
indices n+{A) / n^(A) each boundary relation T : Sj"^ ^ H^ for A* in the sense of [5] satisfies the 
equality dimH = oo (sec [6, Proposition 3.2]). This assertion shows that in the case n_|-(v4) ^ n^{A) 
our definition of a boundary relation is more natural and convenient for applications. Observe also that 
other generalizations of boundary triplets can be found e.g. in [3] . 

Next by using the concept of a boundary relation we investigate in the paper linear relations induced 
by a general (not necessarily definite) canonical system of differential equations with possibly unequal 
deficiency indices. Such a system is of the form 

(1.9) Jv'{t)-B(t)y(t) = l^{t)f{t), t€l, 

where J is an operator in the finite-dimensional Hilbert space H such that J* = = — J and B{t) and 
A{t) are locally integrable [H]-valued functions defined on an interval I = (a, b), — oo < a < 6 < oo, and 
such that B{t) = B*{t) and A{t) > a.e. on X. Without loss of generality we assume that 

(1.10) m = H®H®H 

with the Hilbert spaces H and H and the operator J is 

/ -Ih\ 

(1.11) J= il^ \: H®H®H^H®H®H. 

\Ih / 

The canonical system (1.9) is called Hamiltonian ii H = {0}, in which case the operator J takes the form 




: H ® H ^ H ® H. 



Clearly, the Hamiltonian system is a particular case of the system (1.9). 

Denote by £^ (T) the semi-Hilbert space ofH-valucd Borcl functions f{t) onXwith j.j-{A{t)f{t),f{t))dt < 
oo and let (/, .9)a be the scmi-dcfinitc inner product in £^(X). Assume also that L^(X) is the corre- 
sponding Hilbert space of equivalence classes and n is the quotient map from C j^ (I) onto (I) , so that 
the inner product in Z/i(X) is {f,g){= {■Kf,'Kg)) = {f,g)A, f,g& L\{^)- 
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The nuU manifold M of the system (1.9) plays an essential role in our considerations. Recall [19] that 
J\f is defined as the set of all solutions of the equation Jy'{t) — B{t)y{t) — such that A{t}y{t) = a.e. 
on I. The system (1.9) is said to be definite if A/" = {0} and indefinite in the opposite case. 

As is known the extension theory of symmetric relations is the natural framework for boundary value 
problems involving canonical systems of differential equations (see [27, 21, 9, 10, 14, 2, 22] and references 
therein) . This framework is based on the concept of minimal and maximal relations which are defined as 
follows. Let Tmax be the set of all pairs {y,f} € C\{X) x C\{I) satisfying the system (1.9) and let To 
be the set of all {y, /} e Tmax such that y has compact support. Then Tmax and To are linear relations 
in C\{I) and the Lagrange's identity 

(/, 2;)a - (y, 9)a = [y, Z\b - [y, Z]a, {y, /}, {Z, g} G Tmax- 

holds with 

(1.12) [y,z]a ■.= 'iim{Jy{t),z{t)), [y, z]b '.^ \im{ J y{t), z{t)), y,zGdomTmax- 

tla ffb 

By using (1.12) introduce also the linear relation Tmin in C\il) by 

(1.13) Tmin = {{?/,./} e Tnax ■ [y , z\a = [y , z]b = 0, Z G domT^a^.}. 

Moreover, in the case of the regular endpoint a (that is, if o ^ — oo and B{t) and A(t) are integrable on 

(a,/3), /3eX) let 

(1.14) Ta = {{y, /} e Tmax ■ y{a) = and [y, z]b = 0, 2; e domT;„„^}. 
All the above relations in C\{I) naturally generate by means of the equalities 

(1.15) Tmin = {{Try, tt/} : {y, /} e Tmin), Ta = {{ny, tt/} : {y, /} e Ta}, 

To = {Uy, tt/} : {y, /} e To}, Tmax = {{Try, tt/} : {y, /} e Tmax} 

linear relations Tnm, Ta, Tq and Tmax in the Hilbert space L\{I). 

As was shown in [27] (see also [22, 2]) in the case of the definite system (1.9) Tq is a symmetric linear 
relation in L^(I), Tmin is closure of Tq and Tmax = ^mm(= ^0 ); rnoreover, if the endpoint a is regular 
then Tmin — Ta- In view of this assertion Tmin and Tmax are called minimal and maximal relations 
respectively, which is in full accord with similar definition of minimal and maximal operator for an 
ordinary differential expression [26]. At once certain difficulties arise in the case of an indefinite system 
(1.9), which can be explained as follows. In the definite case the quotient mapping tt isomorphically 
maps domTrnox onto doTaTmax, which enables one to identify in fact the relations Tmax and Tmax- 

If 

the system is indefinite, then the mapping tt I" domTmax has as nontrivial kernel the null manifold M, so 
that the immediate identifying of Tyiax and Tmax becomes impossible. 

The above difficulties were partially overcome in the papers by Kac [17, 18] (the case dim HI = 2) and 
Lesch and Malamud [22] (the case dimH = n < 00), where general (not necessarily definite) systems were 
studied. In these papers first the equality Tq = Tmax is proved and then the minimal relation is defined 
as closure of Tq. 

In the present paper we show that for the general system (1.9) the minimal relation in L\{X) can 

be also defined by the first equality in (1.15) with Tmin in the form (1.13). Moreover in the case of the 
regular endpoint a the minimal relation coincides with the relation Ta defined by (1.14) and the second 
equality in (1.15). Observe also that Ta C Tmin and an interesting in our opinion fact is that generally 

speaking Ta 7^ Tnin (for more details see Proposition 4.13 and Example 4.14). 
Next assume that a is a regular endpoint for the canonical system (1.9), 

:= dimker {ij - /)(== dimi7), := dimker (ij + /)(= dim(ff ® H)) 

and let vi,^ and be indices of inertia of the skew-Hermitian form [y, z]b (for simplicity assume that 
1^6+ > 2^6-). The equality Tmin = Ta enables us to describe all proper extensions of Tmin in terms of 
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boundary conditions. For this aim we use a special boundary relation for Tmax which we call decomposing. 
This boundary relation is defined as follows. 

Let Hb and Hb be finite-dimensional Hilbert spaces and let 

(1.16) Tfa = {Fob : f 6 : T^bV ■ domTmax ^Hb®Hb®Hb 

be a surjective linear map such that 

(1-17) [y, z]b = i{tby, tbz) - {TibV, ^obz) + {TobV, ^ibz), V,ze A.omTmax- 

(it is not difficult to prove the existence of such a map Tb). Moreover, for each function y e domT^a^: let 

(1-18) y{t) = {yo{t), m, yiit)} 

be the representation of y{t) in accordance with the decomposition (1.10). Then the decomposing bound- 
ary relation T : (L^(I))^ Ho (BV-i for T„iax is of the form 

where Ho and Hi are Hilbert spaces defined by means of IS.,Hb and Hb and T'^ : domT^a^ Hj, j G 
{0, 1} are linear maps constructed with the aid of y{a) and the operators from (1.16). If Tmin has equal 
deficiency indices n+{Tmin) = n-{Tmin), then Ho = Hi = H (B H (B Hb and the decomposing boundary 
relation (1.19) can be written as 

In the case of the regular system one can put in (1.20) Toby = yo{b), ^iby = yi{b) and Tby = y{b)- If b 
is not regular, then Tby can be represented by means of certain limits at the point b associated with the 
function y e domT^o^, (for more details see Remark 5.2 ). Therefore the decomposing boundary relation 
r is given by (1.20) in terms of boundary values of the function y € Tmax at the endpoint a (regular 
value) and b (singular value), which is of importance in our considerations of canonical systems. 
Recall [19] that the formal deficiency indices of the system (1.9) are defined via 

N± = dim{y G C\{T) : Jy'{t) - B{t)y{t) = XA{t)y{t) a.e. on I}, A e C±. 

As was shown in [22] the relations 

N+ = n+{Tmin) + k^-, N_ = n- {Tmin) + % 

hold with k^j- = dim A/'. In the present paper by using just a fact of existence of a decomposing boundary 
relation we prove the equalities 

(1.21) N+ = v+ + Vb+, N_=v_+ Vb- ■ 

Formula (1.21) yields the known estimates v± < N± < dimH obtained by analytic methods in[l, 19]. 
Observe also that in a somewhat different way the equalities (1.21) were proved for definite systems in 
[2, Lemma 4.15]. 

Existence of the nontrivial multivalued part mulF of the decomposing boundary relation (1.19) is 
caused by the nontrivial null manifold Af, which can be seen from the equalities 

(1.22) mulF = {{F^?y, T[y} : y e TV}, dim(mulF) = kj^{:^ dim A/"). 

Formula (1.22) implies that for the definite canonical system (1.9) the decomposing boundary relation 
turns into the decomposing boundary triplet 11 = {Ho ® ■Hi,ro,ri} for Tmax- In the case n+{Tmin) = 



6 



VADIM MOGILEVSKII 



n- (Tmin) this triplet is of the form 11 = {H, Tq, Fi} with the boundary Hilbert space H = H (B H (BTib 
and the operators Tj given by 

(1.23) To{y, 7} = {yo{a), ^{y{a) - hy), TobyHe H®H® Ub), 

(1.24) Y^{yJ} = {yM, ^{y{a)+hy),-Tiby}{&H®H®'Hb), {yJ}eTmax 

(here Fob, Fk, and F^ are taken from (1.16)). In the case of the regular system one can put H = H(BH(BH 
and 

(1.25) ro{y, /} = {yo{a), -^{y{a) - y{b)), yo(&)}(e H®H®H), 

(1.26) V^{yJ} = {y^{a),^{y{a)+y{h)),-yi{h)){€H®H®H), {yJ}eTma.- 

The boundary triplet {'H,ro,ri} defined via (1.25) and (1.26) is similar to that introduced, in fact, by 
Rofc;-Bc;kctov [28] for regular differential operators of a higher order. Observe also that other constructions 
of a boundary triplet for Tmax in the case of the definite system (1.9) can be found in [2]. 

The decomposing boundary triplet (1.23), (1.24) enables us to describe maximal dissipative, maxi- 
mal accumulative and self-adjoint boundary conditions, which define in terms of boundary values the 
extensions A D Tmin of the corresponding class. As a consequence we obtain the known description of 
self-adjoint boundary conditions, given in [1, 12, 27] for regular definite systems (1.9) and in [20] for 
definite Hamiltonian systems with the regular cndpoint a. 

Finally by using the concept of a decomposing boundary triplet we examine separated boundary con- 
ditions of various classes. Recall that self-adjoint separated boundary conditions for definite Hamiltonian 
systems were studied with the aid of analytic methods by many authors (sec [15, 20] and references 
therein). In the present paper we show that self-adjoint separated boundary conditions for the definite 
canonical system (1.9) exist if and only if this system is Hamiltonian. Moreover, for the Hamiltonian 
system the decomposing boundary triplet {■H,ro,ri} for Tmax takes the form 

ro{^,/} = {yo{a), roby}{&H®nb), My J} = {yi(a), -TnyK^ H ®nb), {yj} e Tmax 
and the general form of self-adjoint separated boundary conditions is 

(1.27) A = {{y, /} : iVoayo(a) + Ni^yi{a) = 0, NobToby + N^Tuy = 0}, 

where the operators A^oo, -Aio and A^obi ^ib are entries of the self-adjoint operator pairs {(Ao,,. -Aio)} 
and {(A''o;,, iVi;,)}. These results are generalizations of those obtained by Rofe-Beketov in [28] for regular 
differential operators. Moreover, formula (1.27) includes as a particular case the results on self-adjoint 
separated boundary conditions in [15, 20]. 

An interesting in our opinion fact is the existence of maximal dissipative an accumulative separated 
boundary conditions for the not necessarily Hamiltonian system (1.9) (in the paper we describe all these 
conditions) . An important subclass of maximal dissipative (accumulative) separated conditions are those 
defined by a self-adjoint condition at the regular endpoint a and the maximal dissipative (accumulative) 
boundary condition at the singular endpoint b. This subclass of boundary conditions may be useful in 
the theory of not orthogonal spectral functions associated with the system (1.9) (we are going to touch 
upon this subject elsewhere). 

2. Preliminaries 

2.1. Linear relations. The following notations will be used throughout the paper: j^, denote Hilbert 

spaces; [Hi,H2] is the set of all bounded linear operators defined on Hi with values in H2', [H] := [H,7{\; 
A \ £ is the restriction of an operator A onto the linear manifold £; Pc is the orthogonal projector in Sj 
onto the subspace C C Sj; C+ (C_) is the upper (lower) half-plain of the complex plain. 
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Recall that a linear relation T from a linear space Lq to a linear space Li is a linear manifold in 
the Cartesian product Lq x Li. It is convenient to write T : Lq ^ Li and interpret T as a multi- 
valued linear mapping from Lq into Li. li Lq = Li =: L one speaks of a linear relation T in L. For a 
linear relation T : Lq ^ Li we denote by domT, ranT, kerT and mulT the domain, range, kernel and 
the multivalued part of T respectively. The inverse is a linear relation from Li to Lq defined by 
T-' = {{f',f}:{f,f'}€T}. 

Assume now that Ho and Hi are Hilbert spaces. Then the linear space Ho x Hi with the inner product 
{{f,f'},{9,9'})-Ho®-Hi = {f,9)Ho + {.!'■• 9')hi is a Hilbert space^Ho ®Hi. The set of all closed linear 
relations from Ho to Hi (in H) will be denoted by C{Ho,Hi) {C{H)). A closed linear operator T from 
Ho to Hi is identified with its graph grT e C{Ho,Hi). For a linear relation T : Ho ^ Hi we denote by 
T*(e C{Hi,Ho)) the adjoint relation 

In the case T £ C{Ho,Hi) we write e p{T) if kerT = {0} and ranT = Hi, or equivalently if 
T^^ e [Hi, Ho]', G p{T) if kerT = {0} and ran(T) is a closed subspace in Hi. For a linear relation 
T e C{H) we denote by p{T) {A G C : £ p(T- A)} and p{T) = {A G C : £ p(T - X)} the resolvent 
set and the set of regular type points of T respectively. 

For a linear relation T G C{H) and for any A G C wc let 

mx{T) := kcr (T* - \) H Q ran (T - A)), mx{T) := {{/, A/} : / G mx{T)} C T*. 

If A G p{T), then 9Ta(T) is a defect subspace of T. Recall also the following definition. 

Definition 2.1. A holomorphic operator function $(•) : C \ M — >■ ["H] is called a Nevanlinna function if 
lm.z- lm^{z) > and ^*{z) = ^{z), ^ G C\K. 

2.2. Operator pairs. Let K,Ho,Hi be Hilbert spaces. A pair of operators Cj G \Hj,K], j G {0,1} is 
called admissible if the range of the operator 

(2.1) C = {Co ■■ Ci) : Ho (S Hi IC 

coincides with JC. In the sequel all pairs (2.1) arc admissible unless otherwise stated. 

Two pairs {cf ^ : c'f '') : Ho ® Hi ^ Kj, j G {1,2} will be called equivalent if C^^^ = XC'^^ and 
Cf' = XCf ' with some isomorphism X G [/Ci,/C2]. 

It is clear that the set of all operator pairs (2.1) falls into nonintersecting classes of equivalent pairs. 
Moreover, the equality 

(2.2) 9 = {{Co,Ci);IC} := {{ho, hi} e Ho d) Hi : Coho + Cihi = 0} 

establishes a bijective correspondence between all linear relations 6 G C{Ho,Hi) and all equivalence 
classes of operator pairs (2.1). Therefore in the sequel we identify (by means of (2.2)) a linear relation 
9 G C{Ho,Hi) and the corresponding class of equivalent operator pairs Cj G [Hj,IC], j G {0, 1}. 
Next recall some results and definitions from our paper [24] . 

Assume that Ho is a Hilbert space, Hi is a subspace in Ho, H2 ■= HoQHi and Pj is the orthoprojector 
in Ho onto Hj, j G {1,2}. For an operator pair (linear relation) 6 = {(Co, Ci); /C}(g C{Ho,Hi)) we let 

5, := 2Im(CiC*i) - Co2Co*2, Se G [JC], 
where Coi and C02 are entries of the block representation Co — (C02 : Coi) : H2 ®Hi ^ JC. 

Definition 2.2. [24] An operator pair (linear relation) 9 = {(Co, Ci); /C}(g C{Ho,Hi)) belongs to the 
class: _ 

1) Dis{Ho,Hi), if S'e > and G p(Coi - iCi); 

2) Ac{Ho,Hi)), if Se^ and G p{Co + iCiPi); 

3) Sym{Ho,Hi)), iiSe = Q and G p(Coi - iCi) U (Co +iCiTi); 

4) )Self {Ho, Hi), nee Dis{Ho, Hi) n Ac{Ho,Hi)). 
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Note that the inclusion € /o(Coi — iCi) U p{Co + iCiPi) implies that ran (Co : Ci) — JC. Therefore 
each of the above definitions 1) - 4) gives an admissible operator pair (Co : Ci). 

If ^ e Dis{'Ho,'Hi) [9 e Ac('Ho, "Hi)), then dim/C = dim 'Hi (resp. dim/C = dim Ho)- Therefore for an 
operator pair 9 G Dis{'Ho,'Hi) {9 e Ac{'Ho,'Hi)) one can put /C = 'Hi (resp. JC = 'Ho). 

In the case "Ho = "Hi := "H we let Dis{%) = Dis{'H,'H) and similarly the classes AcCH), Sym{J-L) and 
SelfiH) arc defined. In view of Definition 2.2 for each operator pair (linear relation) 9 = {{Co, Ci); A^}(€ 
C{'H}) the following equivalences hold: 



(2.3) 9eDis{n) ^ (Im(CiCo)>0 and € p(Co - iCi)) 

(2.4) 9 e Ac{n) (Im(CiC*) < and e p{Co + iCi)) 

(2.5) 9 e Sym{H) <^ (Im(CiCo*) = and e p(Co - iCi) U p(Co + iCi)) 

(2.6) 9 e Self{H) <^ (Im(CiC^) = and e p(Co - iCi) D p{Co + iCi)) 



Moreover, the classes Dis{%), Ac{H), Sym{H) and Self{H) coincide with the known classes of all 
maximal dissipative, maximal accumulative, maximal symmetric and self-adjoint linear relations in "H 
respectively. 

The following proposition is immediate from Definition 2.2. 

Proposition 2.3. 1) In the case dim'Ho < oo the class Self {Ho, Hi) is not empty if and only if Ho = 
Hi ='. H. 

2) Let dlmH < oo and let 9 = {(Co, Ci); /C}(e C{H)) be an admissible operator pair such that dim/C = 
dim'H. Then the following equivalences hold 

9 e Dis{H) ^ (Im(CiCo*) > 0, 61 e Ac{H) ^ (Im(CiC^) < 0, 9€ Self{H) <^ (Im(CiC^) = 0. 

2.3. Boundary triplets and Weyl functions. Let A e C{Sj) be a closed symmetric linear relation in 

the Hilbert space S) and let n±{A) := dim9lA(^), ^ € C± be deficiency indices of A. Denote by ExtA 
the set of all proper extensions of A, i.e., the set of all relations A G C{Sj) such that A C A C A* . 

Next assume that Ho is a Hilbert space, Hi is a subspace in Ho and H2 ■= Ho Q Hi, so that 
Ho=Hi® H2. Denote by Pj the orthoprojector in Ho onto Hj, j G {1, 2}. 

Definition 2.4. [25] A collection H = {Hq © ?^i,ro,ri}, where Tj : A* Hj, j G {0, 1} are linear 
mappings, is called a boundary triplet for A* , if the mapping F : / — ^ {^of, ^if} from A* into Ho © Hi 
is surjective and the following Green's identity 

(2.7) {f',g) - (/, g') = {tJ, rog)no " i^of, riff)«„ + i{P2rof, P2rog)n, 
holds for all / = {/, /'}, g = {g,g'} e A*. 

In the following propositions some properties of boundary triplets are specified (see [25]). 
Proposition 2.5. // 11 = {"Ho © Hi, To, Fi} is a boundary triplet for A* , then 

(2.8) dim^i = n-{A) < n+{A) = dim "Ho- 

Conversely for any symmetric linear relation A € C{Sj) with n-{A) < n+{A) there exists a boundary 
triplet for A* . 

Proposition 2.6. Let H = {Ho ©"HijFojFi} be a boundary triplet for A* . Then: 

1) kerFo H kcrFi = A and Tj is a bounded operator from A* into Hj, j € {0, 1}; 

2) the set of all proper extensions A G ExtA is parameterized by linear relations (operator pairs) 
9 = {{Co,Ci);K,}. More precisely, the mapping 

(2.9) 9^Ae:= {/ G A* : {Tof, tJ} g 9} 
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establishes a hijective correspondence between the linear relations 9 G C{T-Lo,l-Li) and the extensions A = 
Aq e Ext A- If is given as an operator pair = {(Co, Ci); /C}, then Ag can be represented in the form 
of an abstract boundary condition: 

(2.10) Ae = {/ e A* : CoTof + CiVj = 0} 

3) the extension Ag is maximal dissipative, maximal accumulative, maximal symmetric or self-adjoint 
if and only if 6 belongs to the class Dis, Ac, Sym or Self {Ho, 'Hi) respectively; 

4) The equality 

(2.11) ^o:=kerro = {/e^* :ro/ = 0} 
defines the maximal symmetric extension Aq e Ext a such that n_(^o) = 0. 

It turns out that for every A e C+ (z G C_) the map Tq \ ^xiA) (resp PiTq \ ^ziA)) is an 
isomorphism. This makes it possible to introduce the 7-fields 7n+(') : C+ — > ["Hc-Q], 7n-(') '■ C_ — ^ 
[Hi,^] and the Weyl functions Mn+(-) : C+ ^ [Ho, Hi], Afn-(-) : C_ ^ [Hi, Ho] by 

(2.12) 7n+(A) =7ri(ro f^A(A))"\ A e C+; -rn-{z) = MP^To \ m,{A))-\ z e C_, 

(2.13) Ti \ yix{A) = Mn+(A)ro [ ^x{A), A e C+, 

(2.14) (Fi + zPaFo) f m,{A) = Mn-{z)PiTo \ *Tl.(A), z e C_. 

(here tti is the orthoprojection mSj®^ onto © {0}). According to [25] all functions 7n± and Mn± are 
holomorphic on their domains and (Mn+(A))* = Mn-(A), A e C+. 

Remark 2.7. In the case Ho = Hi := H Definition 2.4 coincides with that of the boundary triplet 
(boundary value space) 11 — {H, To, Fi} for A* given in [13]. For such a triplet n+{A) = n-{A) = dimH, 
Ao{= kerFo) is a self-adjoint extension of A and the relations 

(2.15) 7n(A) = ^i(Fo [ mx{A))-\ Fi [ mx{A) = Afn(A)ro [ ^xiA), A e p{Ao) 

define the 7-field 7n(-) : p(^o) — > [H,i3] and the Weyl function Mn(-) : p{Ao) — > [H] [7] associated with 
operator functions (2.12)-(2.14) via 7n(A) = 7n±(A) and Mn(A) = Mn±(A), A e C±. 

3. Boundary relations and their Weyl families 



Let and Ho be Hilbert spaces, let Hi be a subspace in Ho, let Ho = Hi ® H2 be the corresponding 
orthogonal decomposition of Ho with H2 := Ho Hi and let Pj be the orthoprojector in Ho onto 
Hj, j G {0,1}. In the sequel we deal with linear relations from S)^ into Ho ® Hi. If F is such a 
relation, then an element (f G T will be denoted by <^ = {/, h}, where / = {/, /'} € (/, /' G S)) and 
h = {ho, hi} e Ho ® Hi {ho € Ho, hi e Hi). In this case it will be convenient to write 

(3.1, ,.{/,(^.)}.|(/),(V 

If in addition Hj is decomposed as Hj = Hji ® Hj2 © • • ■'Hj^nj, j S {0, 1}, then the equality (3.1) will 
be also written as 

'{hoi, ho2, . . . , ho,noV 
y{hii, hi2, . . . , hi^m} / 
where hok = Puokho and hik = P-Uikhi are components of ho and hi respectively. 

For a linear relation F : ij^ — )• Ho © Hi its multivalued part is a linear relation from Ho into Hi given 

by 

muir = |{/io,/ii} eHo©Hi : |o, Ml eF 
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Using muir wo define linear manifolds /Cp and K!q in Hi via 

(3.2) KJ^ = Pi dom (mulT) = |a;' e : |o, ^ | e T for some /i2 S and /ii € -Hij , 

(3.3) /C;^ = mul (muir) = jfc" e "^i : |o, } ^ ^} " 

Next introduce the signature operators 

(3.4) Ji5 = "q^^^ : i3 ® i3 ^ i3 e i3, Joi = [jp^ -.Ho® Hi ^ Ho® Hi 

and denote by {S)'^,J^) and ("Ho ® "Hi, Joi) the corresponding Krein spaces. Recall [29] that a linear 
relation T : Sj^ ^ Ho ®Hi is called an isometric relation from (i3^, Jj,) into ('Ho © 'Hi, Joi) if 

(3.5) {Jsjf,g)sj2 = {Joih,x)no®ni, {f,h}, {g,x} e T 
or, cquivalently, if the identity 

(3.6) {f\9)s) - {f,9')f) = {hi,xo)-Ho - {ho,xi)-Ho + i{P2ho, P2Xo)-H2 
holds for every {(/,), (;:»)}, (Z)} e T. 

Definition 3.1. Let A be a closed symmetric linear relation in Sj, let Ho be a Hilbert space and let "Hi 
be a subspace in Ho- A linear relation T : ij^ ^ 'Ho ® 'Hi is called a boundary relation for A* if: 

1) domT is dense in A* and T is an isometric relation from {Sj'^,Jsj) into ('Ho ® "Hi, Joi), i.e., the 
abstract Green's identity (3.6) holds; 

2) if ^ = {(^f,), (2)} GSj''® {Ho ® Hi) satisfies (3.6) for every {(/,), 0} e T, then ipeT. 

The conditions 1) and 2) of Definition 3.1 imply that the boundary relation F is a unitary relation 
from (ij^ Jsj) to {Ho ® Hi,Jqi) [29]. Therefore T is closed and kerT = A. 

Definition 3.2. The families of linear relations M+(A) iHo^Hi, A e C+ and M_{z) :Hi^Ho, zG 
C_ given by 

(3.7) M+(A) = |{/io,/ii}e-Ho©-Hi:|(^^^^^,(^J^"^|Gr forsome /eijj, A e C+; 





[ho 




[hi 




[ho 




[hi 



(3.8) M-{z) = |{Pi/io, hi + iP2ho} : | (^^'^j , l^^^j | G T for some / G | , z G C_ 

are called the Weyl families corresponding to the boundary relation F : i^^ — > 'Hq 'Hi for A* . 

If M+(-) (resp. M_ (•)) is operator- valued, it is called the Weyl function corresponding to the boundary 
relation F. 

In the sequel we deal with boundary relations of the special form introduced in the following proposi- 
tion. 

Proposition 3.3. Assume thatU = {JCo® ICi,Go,Gi} is a boundary triplet for A* (see Definition 2.4), 
/C2 := /Co Q ICi, K-' and K," are Hilbert spaces and 

(3.9) 'Hi :=/Ci®/C'®/C", Ho := /Co © /C' © /C"(= /C2 © Hi). 
Moreover, let Fq e [/C',/Co], F' G [/C'j, let 

(3.10) Fo = (F2 Fi)"^ : /C' /C2 © /Ci 
be the block representation of Fo and let 

(3.11) F' - {F'y + iF2*F2 = 0. 
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Then the equality 

\\ \{G,f + Fik',F;Go} + F'k',k"]) \ j 

defines the boundary relation T : Sj"^ ^ Ho (B Hi for A* such that /Cp = /C' and /Cp = /C". 

Proof. It is easily seen that the following assertion (a) is valid: 
(a) an element ip = {g, (^°)} e ij^ 8 {Ho ® Ui) with 

(3.13) g = {g,g'}&Sf, xo = {mo, x'^, x'q) & Kq ® K' ® K" , xi= {rm, x\, x'l) &Ki®K' ®K" 
satisfies the identity (3.6) for every { (^,) , (^°) } e F if and only if Xq = and the following equalities hold 

(3.14) {f',g)-{f,g') = 

= (Gi/,mo)x;o + {F^Gof,x'o)K' - (Go/,mi)x;o + i(P;c.Go/>x;.mo);c„ / = {/,/'} e A*; 

(3.15) {Fok', mo)Ko + {F'k', x'o)^, - {k', x[))C' =0, k'eK'. 

Let {g, (^°)} e r, so that g € A* and xq, xi are given by (3.13) with 

mo = Gog - iF2x'o, Xq =0, rm = dg + FiXq, x[ = F^Gog + F'x'o- 

Then substitution of such mo, mi and x'l into (3.14), (3.15) and the immediate calculation with taking 
(3.11) and (2.7) into account show that the equalities (3.14) and (3.15) are satisfied. Therefore by 
assertion (a) the identity (3.6) holds for every {(p, (|^°)}, {(^s,), (^o)} g r. 

Assume now that an element {g, (^") } & Sy^ ® {Ho ®Hi) given by (3.13) satisfies the identity (3.6) for 
every {(/,), e T. Then by assertion (a) x'^ = and the equalities (3.14), (3.15) are fulfilled. 

If ./ = {/. /'} e A then Go/ = Gif = and by (3.14) {f',g) - {f,g') = 0. This implies that g G A*. 
Next, in view of (3.10) F^Gof = ^2*^^:2 Go/ + FiPjc^Gof and the equality (3.14) can be written as 

(3.16) {f',g)-if,g') = 

= (Gi/, P^.mo) - (P^.Go/, mi - F^x'^) + i(Pk.Go/, P^.mo + 1^2^^), / = {/, /'} S A*. 
Since the map G = (Gq Gi)^ is surjective, it follows from (3.16) and (2.7) that 

PjCimo = PKiGog, mi ~ Fix'o = Gig, Pfc^mo + iF2x'o = PK^Gog 
and, consequently, 

mo = Gog - iF2XQ, mi = Gig + Fix'q. 

Moreover, by using first (3.15) and then (3.10), (3.11) one obtains 

x[ = F*mo + {F'yx'o = F*Gog + {{F')* - iF*F2)x'o = F*Gog + F'x'^. 

Thus {g, (^°)} G r and the linear relation (3.12) satisfies both conditions of Definition 3.1. 

Finally the equalities /Cp = K' and /Cp = K" are immediate from (3.12). □ 

Proposition 3.4. Let under the assumptions of Proposition 3.3 K" = {0}, so that 

(3.17) -Hii^/de/C', -Ho :=/Co®/C'(=/C2®'Hi). 
and the boundary relation T : Sj'^ ^ Ho ® Hi for A* is (c.f. (3.12)j 
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Assume also that 7n±(') and Mn±(-) are the ^-fields and Weyl functions corresponding to the boundary 
triplet n (see (2.12)-(2.14)J and Fq : ij^ — >■ Ho is the linear relation given by 

To = P«oe{o}r = |{/, ho} GSj^(Bno: |/, Q") | e T /or some /ii e ^Wij . 

Then: 1) kerFo = kerGo, so that the equality 

(3.19) ^0 := kerFo = {/ € A* : {/, 0} € Fq} 

gives the maximal symmetric extension Ao G Ext a with n-{Ao) = ( <s=^ C+ C p{Ao)); 
2) the equalities 

(3.20) 7+(A) = (Fo r ^a(A))-\ 7+(A) = 7ri7+(A), A e C+; 

(3.21) ^-{z) = {PiTo\^M)r\ 7-W = 7ri7-(^), ^ € C_, 

define the holomorphic operator functions ('y-fields) 7+(-) : C+ — >■ ['Hojij] and 7-(-) : C_ [Hi,Sj]. 

Moreover, 

(3.22) 7+(A)=7n+(A)(/Ko zF2), AeC+; -f.{z)=-fu-{z)PK^, z€C- 
and the following identities hold 

(3.23) 7+(m)=7+(A) + (m-A)(Ao-m)"V+(A), A,/ieC+ 

(3.24) 7_(a;)=7_(^) + (a;-^)(AS-a;)-S-(^), 2,weC_ 

(3.25) 7-(^)A = 7+(A) + (^ - A)(^S - '2)"S+(A), A e C+, ^ e C_. 

5^ the corresponding Weyl families are the holomorphic operator functions M+(-) : C+ [Ho, Hi] and 
M_(-) : C_ — >■ [Hi, Ho] associated with Mn±(-) by 

(3.26) M+(A) = (^"^i^^ ^^i+zMn+(A)F2^ ^ ^ 

and M_(2;) = M^(;z), z G C_. Moreover, the block representations 

(3.27) M+(A) = {N+{X) M(A)) ■.K2®Hi^Hu A e C+ 

(3.28) M_(0) = (iV_(2) M(^))"r :Hi^K2®Hu z&C_ 

(3.29) (A) J^g^ : /C2 e -Wi ^ /C2 e -Wi , A e C+ 

(3.30) (^) = ^M{z)) ■^2®Hi^K:2®Hi, z G C- 

induce the Nevanlinna operator functions M(A) (e [Hi]) and M.{X) (g [Hq]); 
4) the following identity holds 

(3.31) M{|I)-M*{X) = {^l-X)^l{X)^+{^l), /x,AgC+. 

Proof. The statement 1) is immediate from (3.18) and Proposition 2.6, 3). 
2) It follows from (3.18) that 

Fo \mx{A) = {{h,{Gofx-iF2k',k'}}:he^x{A), k' G IC'}, A e C+ 

PiFo r ^z{A) = {{h, {PiGoh, k'}} : h e mM), k' € IC'}, ^ e C_. 

This and (2.12) imply that the equalities (3.20) and (3.21) correctly define the operator functions 7-|-(-) 
and 7-(-) such that (3.22) holds. 

Next, one can prove the identities (3.23) - (3.25) in the same way as in [25, Proposition 3.15]. These 
identities show that the functions 7±(-) are holomorphic on their domains. 
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3)-4). Combining (3.7) with (3.18) one obtains 

Letting liere hg — Gq/a — i-P^fc' and taking (3.11) into account we get 

= { {{Mu+{X)ho + {F, + iMZ{m)k', F*ho + {F'rk'}) ■ ^ ^' ^ ^' 

This equahty is equivalent to (3.26). The identity (3.31) is proved in the same way as similar one in [25, 
Proposition 3.17]. Finally, (3.31) implies that A^(-) and M(-) are Nevanlinna operator functions. □ 

In the next proposition we show that under the condition dim Wo < cxd formula (3.12) gives the general 
form of a boundary relation F : -J> 'Ho © 'Hi for A*. 

Proposition 3.5. Let A he a closed symmetric linear relation in Sj and let F : ij^ — > Ho ® "Hi be a 
boundary relation for A* such that dim 'Ho < oo. Then: 

1 ) n±{A) < oo and dom T ^ A* ; 

2) the subspaces /Cp and /Cp (see (3.2) and (3.3)j are mutually orthogonal and, consequently, the 
decompositions 

(3.32) Hi—tCi^tC'r^fCf, Uq := Ko ® IC'y ® K 

hold with JCi := "Hi G (/Cp ® /Cp) and /Co := 'H2 © /Ci; 

3) for any {/, (J^")} G F the inclusion Hq € /Co ® /Cp is valid; 

4) for any k' € /Cp there exists a unique triple of elements /12 S ^2) ki G /Ci and m! G /Cp SMc/i that 

(3.33) (o.(-""tnier 



fci + m' 

Hence the relation (3.33) defines the operators F2 G [/Cp,'H2], Fi e [/Cp,/Ci] and F' e [/Cp] wa 

(3.34) F2k' = h2, Fik' = ki, F'k' = m' {k' € IC'r). 

Moreover, these operators satisfy (3.11); 

5) there exist linear maps Gj : A* — ICj, j € {0, 1} such that {/Co © /Ci, Go, Gi} is a boundary triplet 
for A* and F admits the representation (3.12) with K,' = /Cp, K" = /Cp and the operator Fq given by 
(3.10). 

Proof. 1) Since A = kerF and dim('Ho © "Hi) < 00, it follows that dim(domr/A) < 00. Hence domF = 
domF = A* and, consequently, n+(A) + n-{A) = dim(A*/A) < 00. 

2) - 3). Let {/, (^°)} e F and fc" e /C{^, so that {0, (^°,)} e F. Then by (3.6) (/io,fc") = 0, which 
implies that Hq G "Ho 9 /Cp. 

Assume now that fc' G /Cp, so that {0, ('"',1^''")} G F with some /12 G 'H2(C 'Ho © /Cp) and /ii G 'Hi. 
Then the above statement gives k' + h2 G 'Ho©/Cr and , consequently, k' G Ho©/Cp. Therefore /Cp _L /Cp. 

4) Let k' e /C^. Then according to (3.2) and the first equality in (3.9) {0, {^,X^'+k")} ^ ^ with some 
/i2 e H2, ki e /Ci, m' G /C^, and fc" G /CJ^, which in view of (3.3) implies that |o, (fc'^+t!^^) | G F. Let us 
show that such /12, ki and m' are unique for a given k' . If |o, (|^^*^^) } ^ with some /i2 G H2, /^'i G /Ci 
and m' G /CL then |o, {,r T.^'!"-} ,^)]• € F and the identity (3.6) yields = i\\h2 - h^]? . Therefore 

/i2 — /i2 = and, consequently, (fci — ki) + (m' — ml) G /Cp. Now the decomposition (3.9) yields 
fci — fci = 0, rh! — ml = 0, so that /12 = /12, ^i = fci and m' = m'. 
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The equahty (3.11) for operators F2 and F' is immediate from identity (3.6) applied to |o, (^j^^+jt^^!/) | 
5) Combining (3.33) and (3.34) with (3.2) and (3.3) one obtains 

(3.35) r„ := {0} e m„ir = {{0, i}^^,^;^^ } : . r,, e } . 
Let T C r be a linear relation from S)"^ to "Hq © 'Hi given by 

(3.36) T = ||/, I e r : /lo e /Co, /ii e /Co e/c^| . 
Then in view of (3.35) and statement 3) T n Too = {0} and the decomposition 

(3.37) r = T + 

is valid. This and the equality domF = A* imply that T is the operator with the domain domT = A* . 
Moreover, applying the Green's identity (3.6) to elements {o, {[FH^^F'k'^) } ^ Tex, and {/, {1^^;^,%) } e 
T one obtains 

= {m',k')K' - {kQ,Fik')Ko - iko,F2k')K, = {m\k')K' - {ko^Fok')^,, k' € IC'^. 
Hence m' = Fq fco and formula (3.36) can be written as 

where Go := Pko®{o}T ^^'^ Gi := P^ojeKi^ linear maps from A* to /Co and /Ci respectively. 
Combining (3.37) with (3.35) and (3.38) we arrive at the representation (3.12) of F. 

Now it remains to show that the operators Go and Gi form the boundary triplet 11 = {/Coffi/Ci, Go, Gi} 
for A*. Applying the identity (3.6) to elements of the linear relation T (see (3.38)) one obtains the Green's 
identity (2.7) for Go and Gi. 

To prove surjectivity of the map G = (Go Gi)""" assume that {mo, mi} € /Co ® /Ci and (Go/, mo) + 
(Gi/,mi) = for all f e A* . Then the element 

(3-39) ^ = |o, (. ^ ^"^^ ^"l°'.'p , n;) I ^ ® ® 

I \{-Pimo, F*{mi+iP2mo), 0}J } 

satisfies (3.14) and (3.15) and by assertion (a) in the proof of Proposition 3.3 (fi satisfies the identity (3.6) 
for all {{j,), 1^°)} G F. Therefore by Definition 3.1 and (3.39) <^ e F^o, which in view of (3.35) gives 
mo = and mi = 0. This implies that ran G = Ho®'Hi- □ 

The following two corollaries arise from Propositions 3.3 and 3.5. 

Corollary 3.6. Let T : Sj"^ ^ Ho ® Hi be a boundary relation for A* with dim Ho < 00 and let 
nr := dim(muir). Then: 1) n^{A) < n^{A) < 00 and 

(3.40) dimHo = n+(.4) + nr, dim-Hi = n_(^) + rip. 

2) in the case mulF = {0} (and only in this case) the relation T turns into the boundary triplet 
for A*. More precisely, i/mulF — {0}, then Tq = P-Ho®{0}^ o-i^<i ^1 = P{o}®Hi^ '^'^^ operators and 
n = {Ho © 'Hi,ro,ri} is a boundary triplet for A* . 

Proof. 1) Let {/Co ® /Ci,Go,Gi} be a boundary triplet for A* defined in Proposition 3.5, 5). Then 
according to (2.8) one has dim /Co = n+{A) and dim/Ci = n-{A). Moreover, (3.35) implies that dim(/Cp© 
/Cp) = nr. This and decompositions (3.32) of Hq and Hi yield (3.40). 
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2) If muir = {0}, then by (3.35) /C[, = /C^^ = {0}. Therefore in view of (3.32) and (3.12) the boundary 
triplet {/Co ® /Ci, Go, Gi} satisfies the equalities ICj = Hj and Gj = Fj, j e {0, 1}. □ 

Corollary 3.7. Assume that A is a closed symmetric linear relation in Sj, "Ho is a Hilbert space with 
dim "Ho < oo, Hi is a subspace in Ho and T : ^ Ho (B Hi is a linear relation such that domF = A* 
and kerF = A. If the identity (3.6) is satisfied for T and 

(3.41) dim{Ho®Hi) = n+iA) + n-{A) + 2nr, 

then r is a boundary relation for A* . 

Proof. Applying the same arguments as in the proof of Proposition 3.5 one obtains decompositions (3.32) 
of Ho and Hi and the equality (3.12) with /C' = /Cp, JC" = /Cp and operators Gj : A* Kj, j G {0, 1}, 
satisfying the Green's identity (2.7). Moreover, by (3.35) dim(/Cp ® /Cp) = nr and (3.32) together with 

(3.41) gives dim(/Co ® /Ci) = n+{A) + n-{A). Observe also that in view of (3.12) ker G = kerF = A (here 
G = (Go Gi)"^) and hence 

dim(domG/ker G) = dim{A*/A) = n+{A) + n-{A) = dim(/Co ® /Ci). 

This implies that ranG = /Co®/Ci and, consequently, the operators Gq and Gi form the boundary triplet 
{/Co ® /Ci, Go, Gi} for A*. Therefor by Proposition 3.3 F is the boundary relation for A*. □ 

In the case Ho = Hi H the above statements on boundary relations can be rather simplified. 
Namely, the following corollary is immediate from Propositions 3.3 - 3.5. 

Corollciry 3.8. Assume that H = {/C, Go, Gi} is a boundary triplet for A* (in the sense of [13]), /C' and 
K" are Hilbert spaces and 

•H := /C ® /C' ® K". 

Moreover, let F G [/C',/C] and F' = {F')* e [/C'] be linear operators. Then the equality 

(3.42) T=llf,(^ ^ {Go/,fc',0}} \\:feA*,k'e}C',k"G]C"] 
^ ' \y'\{Gif + Fk',F*Gof + F'k',k"})\ j 

defines the boundary relation F : — > H^ for A* . 

If in addition /C" = {0}, then the following statements are valid: 

1) the equality (3.19) defines the self-adjoint extension Aq of A and Ao = ker Go; 

2) the relations 

(3.43) 7(A) = (Fo r^lA(A))-\ 7(A) = 7ri7(A), A e p(Ao); 

(3.44) grM(A) = \{h, h'} eH^:l^ (^Q , ^eV for some / e ijj , A e p{Ao), 

define the ^-field -f{-) : p{Ao) — )■ [H,^)] and the Weyl function M(-) : p{Ao) — > [H] corresponding to F. 
Moreover, 7(A) and M{X) are associated with the -y-field 7n(A) and the Weyl function Mn(A) for the 
triplet n (see (2.15)^ via 

(3.45) 7(A) = 7n(A)Px;, M(A) = (^^"1^^ : /C ® /C' ^ /C ® /C', A e p(^o) 
and the following identities hold 

(3.46) 7(m) = 7(A) + (m - X){Ao - /x)-S(A), A e p{Ao) 

(3.47) M(/.) - M*(A) = (/X - A)7*(A)7(m), A e p{Ao). 
The identity (3.47) implies that M(-) is a Nevanlinna operator function. 
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Conversely, let F : Jo^ — )■ be a boundary relation for A* with dim'H < oo and letfC'^ = dom (mulF), /Cp = 
mul(muir) (c.f. (3.2) and (3.3);. Then 

n+ {A) = n_ {A) = dim H — np 

(here nr = dim(mul F) ) and the following statements hold: 

1) H=JC®}C'y®JC'^, where /C = H e (/C^. ® }C'{.): 

2) F admits the representation (3.42) with some boundary triplet H = {/C, Go, Gi} for A* and operators 
F e [IC't,IC] and F' = {F')* e [K't]. 

Remark 3.9. 1) In the case Ho our Definitions 3.1 and 3.2 of the boundary relation F : ij^ ^• 

T-L^ for A* and the corresponding Weyl family Af(-) coincide with that introduced in [5]. 
2) The identities (3.46) and (3.47) were proved in [5] (see also [6]). 

4. Canonical systems 

4.1. Notations. Let I = {a, b) (— oo < a < b < oo) be an interval of the real line (in the case a > — oo 
(resp. b < oo) the endpoint a (resp. 6) may or may not belong to X) and let HI be a finite-dimensional 
Hilbert space. Denote by C]^^iX) the set of all Borel operator functions F{-) defined almost everywhere 
on I with values in [H] and such that J ||-F(t)|| dt < oo for any finite segment [a, 13] C X. 

[«./3] 

Next, denote by AC {I) the set of all functions /(•) : I H, which are absolutely continuous on 

any segment [a, /3] C X. Moreover, let ACo{X) be the set of all functions / G AC{I) with the following 
property: if a G I (resp. 6 S I), then /(a) = (resp. f(b) = 0); otherwise f{t) = on some interval 
(a, a) C I (resp. (/3, b) Cl). Clearly, in the case of a finite segment I = [a, b] the set ACo{I) coincides 
with the set of all functions / € AC{T) such that /(a) = f{b) = 0. 

Let A(-) e C]^^{X) be an operator function such that A(t) > a.e. on X. Denote by C\{X) the 
linear space of all Borel functions fif) defined almost everywhere on I with values in EI and such that 

J {A{t) f (t) , f {t))M dt < oo. As is known [16, 11] C\{X) is a semi-Hilbert space with the semi-definite 

I 

inner product (•, •)a and semi- norm || • ||a given by 

{f,g)A = J^{mm,g{t))Mdt, ||/||a = ((/,/)a)5, f,geCliX). 

The semi-Hilbert space C\{X) gives rise to the Hilbert space L\{X) = C\{X)/{f G C\{X) : ||/||a = 0}, 
i.e., L\{X) is the Hilbert space of all equivalence classes (/ equivalent to g means A{t){f{t) — g{t)) = 
a.e. on X) in £^(I). The inner product and norm in L\{X) are defined by 

(/,?) = (/,5)a, ||/|| = (/,/)^ = ||/||a, IgGLliX), 

where / € / (g € is any representative of the class / (resp. 5). 

In the sequel we systematically use the quotient map tt from Cj^iX) onto L\{X) given by tt/ = /(9 
/), / G £i(J). Moreover, we let 5? := tt © tt : {C\{X)f ^ {L\{X)Y , so that n{f,g} = {f,g}, f,g e 
C\{X). It is clear that kerTr = {/ G C\{X) : A{t)f{t) = a.e. on I}. 

4.2. Minimal and maximal relations. Let as above X = (a, b) (—00 < a < b < 00) be an interval and 
let HI be a Hilbert space with n := dimH < 00. Moreover, let -B(-), A(-) G jCj^^{X) be operator functions 
such that B{t) = B*{t) and A{t) > a.e. on X and let J G [H] be a signature operator ( this means that 

J* = J-i = -J). 

A canonical system (on an interval X) is a system of differential equations of the form 
(4.1) Jy'{t)-B{t)y{t) = A{t)f{t), t G I, 
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where /(•) G £^(X). Together with (4.1) we consider also the homogeneous canonical system 

(4.2) Jy'{t)-Bit)y{t) = XAit)yit), t€l, A e C. 

A function y G AC{T) is a solution of (4.1) (resp. (4.2)) if the equality (4.1) (resp. (4.2) holds a.e. on I. 
In the sequel we denote by J\fx the linear space of all solutions of the homogeneous system (4.2) belonging 

to Clil): 

(4.3) Mx = {y€ AC{T) n C\{X) : Jy'(t) - B{t)y{t) = XA{t)yit) a.e. on I}, A € C. 

It follows from (4.3) that dimWA < dimH < oo. 

As was shown in [19] the set of all solutions of (4.2) such that A{t)y{t) = (a.e. on I) does not 
depend on A. This enables one to introduce the following definition. 

Deflnition 4.1. [19] The null manifold Af of the system (4.1) is the subspace of TVa (A G C) given by 

(4.4) TV = TVa n kerTT = {y € AC{I) : Jy'{t) - B{t)y{t) = XA{t)y{t) and A{t)y{t) = a.e. on I}. 
For each c S I denote by He the subspace 

(4.5) ee = {y(c):y(-)eAA}(cH) 
and let 

(4.6) kj^ = dim = dim He- 
Clearly, AA C AAa (A e C) and for any fixed Aq e C \ M 

(4.7) U = Ux,nAfj^. 

According to [27, 17, 18, 22] the canonical system (4.1) induces the maximal relations Tmax in 'C^(X) 
and Tmax in L\{X), which are defined by 

(4.8) Tmax = {{y, /} G C\{T) X C\{T) : y G AC(X) and Jy'(t) - B{f)y{t) = A{t)f{t) a.e. on T}, 

(4.9) Tmax = T^Tmax = {{v, /} € L\{I) © L\{T) : y = Tiy and / = tt/ for some {y, /} G Tmax}- 
For {y, /}, {z, g} G Tmax and a segment [a, /3] C X integration by parts yields 

/ {A{t)f{t), z{t)) dt- [ {A{t)y{t),g{t)) dt = {Jy{f3), zip)) - {Jy{a), z{a)). 
Hence there exist the limits 

(4.10) [y, z]a := lim( Jy(a), z{a)), [y, z]b := lim(Jy(^), 2;(/3)), y,ze dom7;,iax 
and the following Lagrange's identity holds 

(4.11) (/, 2;)a - {y, g)A = [y, Z]b - [y, z]a, {y, /}, {z, g} G Tmax- 

Formula (4.10) defines the boundary bilinear forms [•, ■]a and [•, •];, on domT^aa;, which play an essential 
role in what follows. By using this form we define the minimal relations Tmin in J~-%{T) and Tmin in 
L\{1) via 

(4.12) Tmin = {{y, /} e Tmax ■ [y, z]a = and [y, z]b = for every z G domTmax}, 

(4.13) Tmin = T^Tmin = {{v, 1} € L%{I) Lj^il) y = Try and / = tt/ for some {y, /} G Tmin}- 
Moreover, introduce linear relations To in and Tq in i^(X) by letting 

(4.14) To = {{y, /} G Tmax ■- y e ACo{I)}, To = nTo- 

It is clear that To C Tmin C Tmax and To C Tmn C Tmax- 

Our next goal is to show that Tmin is a closed symmetric linear relation and T^^^ = Tmax- To do this 
we start with the following definition. 
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Definition 4.2. A finite endpoint a (rcsp. b) of the interval I = {a, b) is said to be a regular endpoint 
of the canonical system (4.1) if a e I (rcsp. b e X). The canonical system (4.1) is called regular if both 
endpoints a and b are regular; otherwise it is called singular. 

Clearly, in the case of the regular endpoint a (resp. b) integrals / ||B(f)|| rft and / ||A(f)|| rft (resp. 

[a,c] [a,c] 

/ \\B{t)\\dt a,nd J ||A(i)|| di) are finite for any c e (a,6). 

[c,b] [c,b] 

If the system (4.1) is regular, then the identity (4.11) can be written as 

(4.15) (/, z)a - {y, g)A = {Jy{b), z{b)) - {Jy{a), z{a)), {y, /}, {z, g} e Tmax- 

In the case of the regular system (4.1) we associate with every subspace K c HI two pairs of linear 
relations 7k, 7k* in and Tk, Tk- in L\{1) given by 

(4.16) Tk = {{y, /} e Tmaa. : y{a) G K and y{b) = 0}, 

(4.17) Tk. = {{y, /} e Tmax ■■ y{a) e ( JK)^}, 

(4.18) Tk = 7r7Kj ^k* = 7''7k*- 

Lemma 4.3. If the system (4.1) is regular, then for any subspace K e HI and A e C 

(4.19) ran (Tk - A) = (ker (Tk* - A))^. 

Proof It follows from (4.16) and (4.18) that ran(TK - A) is the set of all /e L\{I) with the following 
property: there are / € / and y G ACil) such that 

(4.20) y{a) € K, y{b) = and {y, f + Xy} G Tmax- 
On the other hand, (4.17) and (4.18) imply that 

(4.21) ker (Tk* - A) = {z & L\{I) : {z,\z} € Tmax and z{a) € (JK)-^ for some z€z}. 

Let / € ran(TK — A), z G ker (Tjj* — A) and let {y,f + Ay}, {z,\z} be the corresponding elements of 
Tmax from (4.20), (4.21). Applying to these elements the Lagrange's identity (4.15) one obtains 

{f,z)A = -{Jy{a),z{a))=0. 

Hence (/,z) = and, consequently, ran(TK — A) C (ker (Tk* — A))^. 

To prove the inverse inclusion assume that / e (ker (Tk* - A))-^ and let / G /, / S C\{I). Moreover, 
let y e AC{I) be the solution of the equation 

Jy'-B{t)y = A{t){f{t)+Xy) 

with the initial data y{b) = 0, so that {y, f + Xy} G Tmax- Next, for every h € ( JK)-"- let Zh € AC{I) be 
the solution of the equation 

Jz' - B{t)z = XA{t)z 

with the initial data z/i(a) = h and let z/j = irzh- Since {z/i,Az^} G Tmax and Zh{a) € (JIK)-*-, it follows 
from (4.21) that Zh € ker (Tk* — A) and, therefore, {f,Zh) = 0. Now application of the identity (4.15) to 
{y, f + Xy} and {zh,Xzh} gives 

{Jy{a),h) = iJy{a),Zh{a)) - {Jy{b),Zh{b)) = -if,Zh)A = -{Izh) =0, he (JK)^, 

which implies that y{a) G K. Thus for an arbitarry / € (ker (Tk* — A))-"- we have constructed / G / and 
y e AC{I) satisfying (4.20). This gives the requiered inclusion (ker (Tk* — A))-"- C ran (Tk — A). □ 
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Lemma 4.4. // the system (4.1) is regular, then for any subspace K C H 

(4.22) (Tk)*=Tk.. 

In the particular case K = {0} formula (4.22) gives Tq = T^ax- 

Proof. Applying (4.15) to {y, /} € 7ik and {z,g} G 7k* we obtain (/, z)a - {y,9)A = 0. Therefore by 
(4.18) one has Tk* C (Tr)*. 

Let us prove the inverse inclusion. First observe that in view of (4.21) (with A = 0) dimkerTK* < 
dimA/o < oo. Hence kerTx* is a closed subspace in L\{X) and (4.19) gives 

(4.23) kerTfc - (ranTs)^. 

Let {zj} e (Tk)*. Choose f e f, f e jC\{I) and let y e AC{T) by the solution of (4.1) with 
y{a) = 0. Then {y, f} e 7k* and by (4.18) {y, /} e Tk*(c {Tk)*) (here y = ny). Thus {£- y, 0} e (Tk)*, 
which implies that z — y G (ranTK)"*". Therefore by (4.23) z — yG kerTjc*, so that {z — y,0} € Tjj*. Now 
representing {z, /} as 

{lj}^{z-y,0} + {yj} 

and taking into account that both terms in the right hand part belong to Tk* one obtains {z, /} S Tk*. 
This proves the desired inclusion (Tk)* C Tk*. □ 

Lemma 4.5. Let the singular canonical system (4.1) be defined on an interval I = {a, b). For every finite 

segment X' = [a',b'] C T denote by T^^ax '^"'^ "^max maximal relations in C\il') and L\il') respectively 
induced by the restriction of the system (4.1) onto I' . Then there exist a finite segment Xq C I, a 
point c G I'q and a subspace Ho C H with the following property: for any segment I' D Iq and for any 
{y^ /} G "^max there exists a unique function y € AC{I') such that y €y, y{c) G Hq and {y, /} € T^ax 
for any f &f. 

Proof. Fix a point c € T and for any segment T' 9 c put 

(4.24) M^' ={ye AC{I') : Jy'{t) - B{t)y{t) = and A{t)y{t) = a.e. on I'}, 

M^' ={y{c): y{-)eAf^'}. 

Clearly, the inclusion X[ C X'2 yields H-^^ q H-^i. Since dimH < 00, this implies that there exists a finite 
segment X'q C X such that H^' = W^o for all X' D X'q. Put Hq := (H^o)-^ and show that such X'q and Hq 
have the desired property. 

If I' D Tq and {y, /} G T^^^, then according to definition (4.9) there is a function y G AC{X') such that 
y G y and for any / G / the equality (4.1) holds a.e. on X' . Let y € ACiX') be the solution of the equation 
(4.1) on X' with the initial data y{c) = 7ho2/(c)(G Hq) and let (fi = y — y. Then Jf'{t) — B{t)ip{t) = a.e. 
on X' and <fi{c) = y{c) - y{c) e H^. Since H^ = H^o = M^' , it follows that ip{c) e H^'. Hence <fi e M^' 
and, consequently, A{t){y{t) — y{t)) = A{t)ip{t) = a.e. on X'. This means that y G y, y{c) G Hq and 

{yJ}&T^'ax- 

To prove uniqueness of such y assume that z G AC{X') has the same properties, i.e., z ey, z{c) e Hq 
and {z, /} e 7j^aa; ^'^^ f ^ f- Then the function ip := y—z satisfies the equalities J^p' {t) — B{t)ip{t) = 
and A{t)il){t) = a.e. on I'. Hence tp G and, consequently, V(c) € H-^ (= Hq ). On the other hand, 
V'(c) e Ho, so that V(c) = 0. Therefore V' = and hence y = z. □ 

Proposition 4.6. Let Tq be the linear relation in L'^{X) given by (4.14) . Then 

(4.25) To* = Trr 
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Proof. In the case of the regular system (4.1) the equahty (4.25) was proved in Lemma 4.4. 

Assume now that the system (4.1) is singular. Then applying the Lagrange's identity (4.11) to {y, /} € 
Tmax and {z,g} e To we obtain 

(4.26) (./,^)a-(2/,5)a = 0. 
Therefore by (4.9) and (4.14) one has T^ax ^ -^o * 

Let us prove the inverse inclusion Tq c Tmax- Assume that {y, f} € Tg* and choose y,f G ^^i^) 
such that ny = y and nf = f. Next, for every finite segment I' = [a', b'] C I denote by yx' and fx' 
the restrictions of the fimctions y{-) and /(•) onto I' and let yx' = t^i'Vi', fi' = T^X'fx', where ttx' is 
the quotient map from £^(X') onto L\{X'). Assume also that and Tq are linear relations (4.14) in 
£^(I') and L^(X') respectively induced by the restriction of the system (4.1) onto I' . 

Let {zx',,gx'} G and let z{f?) and g{t) (t G X) be functions obtained from zx' and gx' by means of 
their zero continuation onto I. Then {z,g} G To and, consequently, (4.26) holds. Therefore 

/ {A{t)fx'{t),zx'{t))dt- [ {A{t)yx'{t),gx'{t))dt = 0, {zx' , gx'} € T^' , 
Jx' Jl' 

which implies that {yx',fx'} € {Tq )*. Moreover, {Tq )* = T^ax^ because the restriction of (4.1) onto I' 
is a regular system. Thus, {yx', .fi'} G T^ax every finite segment I' C I. 

Next, by Lemma 4.5 there exist a finite segment Xq C X, a point c & I'q and a subspace Hq C H with 
the following property: for any finite segment X' D Xq there exists a unique function yx' € ^C(X') such 

that 'Kx'Vx' = yi') yi'(c) G IKo and {yx'^fi'} S T^l^,- Moreover, by using uniqueness of the function yx' 
(for a given X') one can easily verify that for any pair of segments XJ , Xj such that Xq C Xj C X2 C X the 
restriction y^;/ |" I[ coincides with y^^,^ . This allows us to introduce the function y G AC (X) by setting 

y{t) = yx'{t)i t G I, where X' is an arbitrary segment such that Xq C X' C X and t G I'. It is clear 
that ny = y and {y, /} G T^ox, which gives the inclusion {y, /} G Tmax- Therefore Tg C T^ax and the 
equality (4.25) is valid. □ 

Lemma 4.7. Let the eanonical system (4.1) be given on an interval T = [a, 6) with the regular endpoint 
a. Assume also that 71, T2 and Ti, T2 are linear relations in L\{I) and L\{T) respectively defined by 

(4.27) Ti = {{y, /} G T;„ax : b, z]b = for every z G domT^^ax}, Ti = wTi, 

(4.28) T2 = {{y, /} G r™„x : y(a) = 0}, T2 = ^FTa.. 
Then 

(4.29) Ti* = T2. 

Proof. The inclusion T2 C T* follows from the identity (4.11) applied to {y,f} G Ti and {z,g} G T2- 

To prove the inverse inclusion assume that {y, f} G T* and let y,f € C\{I), iry = y, tt/ = /. 
Moreover, for every /3 G X let X^ := [a, let y/3 and fp be the restrictions of the functions y(-) and 
/(•) onto X^ and let y^g = tt/jj//?, //j = tt/j/^j, where tt/j is the quotient map from C\ilp) onto L^(X^). 
Consider also linear relations , in £^(X^) and T'^, Xf in L^(Xg) given by 

= {{y, /} e T±x ■■ yiP) = 0}, T^ - ^/^r'^ 

(4.30) T;'^ = {{y, /} G r^„, : y{a) = 0}, T^^ = npT^. 

It follows from (4.22) (with IK = H) that {T^)* = and the same arguments as in the proof of 
Proposition 4.6 give the inclusion {yp, fp} G T^, /3 G X. 

Next, according to definition (4.30) of T^ there is a function y^ G AC(X^) such that irpyp = 
yp, y^{a) = and Jy'^{t) — B{t)y^{t) = A{t)f^{t) a.e. on X^. Moreover, it is easily seen that for a 
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given P £ I such a function is unique, so that = ^/Sa t -^Pi ^'^^ /^i < /^2- Therefore the equahty 
y{t) = yp{t), t G I, p > t correctly defines the function y G AC (I) such that ny = y and {y,f} € 72- 

This imphes that {y, J} e T2 and hence Tf c T2. □ 

Proposition 4.8. Let a be a regular endpoint of the canonical system (4.1) and let Ta and Ta be linear 

relations in and L\{I) respectively given by 

(4.31) Ta = {{y, /} e Tmax ■ y{a) = and [y, z]b ^ for every z G dom7^„a2;}, Ta = irTa- 

Moreover, let Tq be the relation (4.14) and let Tq be the closure of Tq. Then Ta is a closed symmetric 
relation and 

(4.32) Ta = To 

(4.33) T^ = Tjnax 

Similarly if h is a regular endpoint of the system (4.1) and 

(4.34) Tb = {{y, f] e Tmax ■ y{b) = and [y, z]a = for every z G doni7;rjax}, Tt = nTb, 
then Tb is a closed symmetric relation in L\{1.) and 

(4.35) Tb = To, T^ =Trnax- 

Proof. Applying the Lagrange's identity (4.11) to {y,/} € Ta and {z,g} € Tmax one obtains the equality 
(4.26). Therefore 

(4.36) Tmax^T: and Ta^T*^,,. 

Moreover, by (4.31) Ta C Tmax, which together with the first inclusion in (4.36) shows that Ta is 
symmetric. 

Next, assume that T and T2 arc the linear relations (4.27) and (4.28). Since T C Tmax, it follows 
that r;^„^ C Tj" and by (4.29) r^„^ c T2. Therefore for any {y, /} G T;^^^ there is {y, /} e Tmax such 
that y{a) = and 7r{y, /} = {y, /}. This and the identity (4.11) give 

[y-,Ab = {f,z)A - {y,g)A =0, ZG domT^^aa:, 

which implies that {y, /} G Ta- Hence T^^^ c Ta and by the second inclusion in (4.36) 
(4-37) Ta=T*^^^. 

Therefore Ta is closed. Moreover, by (4.25) T„iax is also closed, which together with (4.37) gives (4.33). 
Finally, combining (4.25) with (4.33) we arrive at (4.32). 

Similarly one proves the relations (4.35). □ 

Corollary 4.9. Under the assumptions of Lemma J^.l the equality T| = T\ is valid. 

Proof. It follows from (4.33) that for each A e C \ R the defect subspace of the close symmetric relation 
Ta is 

^x{Ta) = ker {Tmax - A) = ttTVa. 

Therefore Ta has finite deficiency indices and (4.27) gives C Ti C Tmax- Consequently, T is closed 
and the required equality T2 = T follows from (4.29). □ 

Lemma 4.10. Let the canonical system (4.1) be defined on an interval I = {a,b). For each subinterval 
Iff := b) {(3 G I) denote by T^ax '^'^^ "^max maximal relations induced by the restriction of the system 
(4.1) onto Ig and let T^ and Tf be linear relations in C\{Ifi) and L\{Iff) respectively given by 

(4.38) T^^ = {{y, /} G Tlax ■■ [y, z]b = for every z G domT^^J, Tf = n^Tf . 
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Then there exists a subinterval (Z I, a point c C 2fj„ and a subspace H C HI with the following 
property: for any interval Ig D X^(, and for any {y, /} e Tf there exists a unique function y € ACil^) 
such that njsy = y, y{c) G H and {y, /} e Tf for any f G f. 

Proof. Fix a point eel and for any interval X^(= [13, b)) B c let 

J\f^ = kerTf nkerTT^j = {y ^ AC{Ip) : {y,0} e Tf and A{t)y{t) = a.e. on I^}, 

= {y{c):yGM^}. 

Let us prove the following assertion: 

(a) if I;3, clp,{^ P2<Pi) and 2/2 G AA'3^ then yi := 1 € A/''^^- 
Indeed, the inclusion y G J\f^ is equivalent to the relations 

(4.39) Jy'{t) - B{t)y{t) = and A{t)y{t) = a.e on Ig, 

(4.40) ]imiJy{t),z{t))=0, {z,g}GTL.- 

Since yi is a restriction of 2/2 and (4.39) holds for j/2 on I''^, it follows that (4.39) is valid for yi on I^'^ . 
Next, assume that {-21,51} € T^^^ ^"^^ 2;(t) be the solution of the equation 

Jz'{t)-B{t)z{t)={), ie[^2,/3i] 

such that z{j3i) = zi{pi). Then the pair {2:2, 52} with 

f.l(t),tel,, ,2(t)=^^^'^''"^^^ 

\^(0, te[/32,/3i) ^ \o, te[/32,/3i) 

belongs to T^o^, and, consequently, \\m{Jy2{t),Z2{t)) = 0. At the same time ^1 = 2:2 \ ^iSi, so that 

limUyiit), zi{t)) = 0. Hence (4.40) holds for t/i, which completes the proof of the assertion (a). 

It follows from (a) that C Xp^ yields H^^ C M^'^. Since dim HI < 00, this implies that there exists 
a subinterval 1^^ = [(3o,b) such that — M./^° for all Ijs D Ifj^. Next by using the same arguments as 
in the proof of Lemma 4.5 one shows that the statement of the lemma holds for the constructed above 
interval I^^ and the subspace H = (H^o)-"-. □ 

Lemma 4.11. Let the canonical system (4.1) be given on an intervall = (a,&). Assume also thatTi, Ts 

and Ti, T3 are linear relations in L\{I) and L\{I) respectively given by (4.27) and 

(4.41) % = {{y, /} e Tmax ■ [y, z]a = for every z G domT^r^ax}, T3 = wTs- 
Then 

(4.42) T3* = Ti. 

Proof. We give only the sketch of the proof, because it is similar to that of Proposition 4.6 and Lemma 
4.7. 

The inclusion Ti C Tg* follows from the Lagrange's identity (4.11). To prove the inverse inclusion 
assume that {y,f} G T3. For every intervall/j = 6) construct the restrictions yjs, fp G L\{Xp) onto 
1/3 in the same way as in the proof of Lemma 4.7. Moreover, let Ti and Tf be the relations (4.38) and 
let T2 = {{y-if} G Tj^ax '■ y{P) = 0}j "^2 = T^p'Ti ■ Then by using the Lagrange's identity one proves 
the inclusion {y^, fp} e {t!^)* . At the same time by Corollary 4.9 {T^)* = Tf , so that {yp, Jp} e Tf 
for every interval Xp. Now by using Lemma 4.10 one obtains the inclusion {y, /} S Ti. Hence T3* C Ti, 
which yields (4.42). □ 

Now we are ready to prove the main theorem of the subsection. 
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Theorem 4.12. Let Tmax and Tmin be maximal and minimal relations (4.9) and (4.13) induced by the 
canonical system (4.1) on the interval X = {a, b) and let Tq be the relation (4.14). Then Tmin is a closed 
symmetric linear relation in L\{^) and 

(4.43) T{) T^nin; ^inin ^'m,ax- 

If in addition the endpoint a (resp. b) is regular andTa (resp. Ti,) is the relation (4.31) (resp. (4.34)^, 
than Tmin = Ta (resp. Tmin = Tb). 

If the system (4.1) is regular, then Tmin = To and every X gC is a regular type point of Tmin, that is 

p(Tmin) — 'C. 

Proof. It follows from the Lagrange's identity (4.11) that Tmax C T^j^ and Tmin C T^^^. This and the 
obvious inclusion Tmin C Tmax show that Tmin is symmetric. 

Next assume that Ti and T3 are the linear relations (4.27) and (4.41). Since T^ C Tmax, it follows that 
Tmax C and by (4.42) T^^^ C Ti. Now the arguments similar to that in the proof of Proposition 4.8 
give the equality T^^^^ = Tmin, which together with (4.25) leads to (4.43). Moreover, combining (4.43) 
with (4.32) and (4.35) wc arrive at the required statement for systems with the regular endpoint a or b. 

Assume now that the system (4.1) is regular and show that in this case 



(4.44) ker (Tq - A) = {0}, ran (Tq - A) ran (Tq - A), A e C. 

If y G ker (Tq — A), then {y, Xy} G Tq and, consequently, there is t/ G AC(I) such that ny = y, y{a) = 
y{b) = and y is a solution of (4.2). Hence y = 0, which gives the first equality in (4.44). Moreover, 
formula (4.19) (with K = {0}) implies the second equality in (4.44). 

Since Tq is symmetric, it follows from (4.44) that Tq is closed. Therefore by (4.43) Tmin = Tq and 

(4.44) yields the equality p{Tmin) = C. □ 

Let N be the null manifold (4.4) of the canonical system (4.1). Then {j/,0} € Tmax for every y G J\f 
and the Lagrange's identity (4.11) gives 

(4.45) [y,z]a = [y,z]b, y G Af, z G domTmax- 
This enables us to introduce the subspacc Af' C Af via 

(4.46) Af' = {y GAf : [y, z]a = 0, zE dom7;„ai;} ^ {y e Af : [y, z]b = 0, z e domT^aa;}- 
Next, the relations {y, /} G Tmax and Tr{y, /} = mean that y G AC{X), f G C\{fL) and 

Jy'{t) - B{t)y{t) = A(i)/(i), A(i)2/(i) = 0, A(i)/(i) = a.e. on I. 

Therefore 

(4.47) ker \ Tmax) = {{y, /} e C\{I) X C\{I) -.yGAf and A{t)f{t) = a.e. on I}, 

(4.48) ker {n \ Tmin) = {{v, /} € Cl{I) x £i(J) : y e Af' and A{t)f{t) = a.e. on I}. 

Proposition 4.13. Let a be a regular endpoint of the system (4.1), let Ta be the linear relation (4.31) 
and let Af' = {{y, 0} : y € Af'}. Then 

(4.49) Tm^n=Ta+Af', 

which implies that the equality Tmin = Ta holds if and only if Af' = {0}. 

Proof. Since Ta C Tmin and by Theorem 4.12 nTmin = T^Ta{= Tmin), it follows that 

(4.50) Tmin =Ta+ kcr (tT \ Tmin)- 

Clearly, the inclusion {0, f} ^Ta holds for any / G 'C^(X) with A{t)f{t) = a.e. on X. Combining this 
assertion with (4.50) and (4.48) one obtains Tmin = Ta +Af' ■ Moreover, for each y G jVndom?^ one has 
y(0) = , so that y = 0. Hence Ta f\Af' = {0}, which gives the direct decomposition (4.49). □ 
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Example 4.14. Consider the canonical system (4.1) with H = C'^ and operator coefficients J, B{t) and 
A{t) given in the standard basis of by the matrices 

-^=(1 ~o)' ^(*)=0' ^W=(?) 2)' *e[0,oo). 

One immediately checks that for this system Af\ — Af — {y{t) = {0, C} : C € C} and each function 
^; G dom 7^02: is of the form 2(f) = {0, Z2{t)}{&C^). Hence {Jy{t), z{t)) = {y&Af, z € domT^aa;), so 
that M' = N ^ {0}. This example shows that there exist canonical systems with the regular endpoint a 

such that Tmin 7^ Ta- 

4.3. Deficiency indices and Neumann formulas. Let %nax be the maximal relation (4.8) in C\{I) 
induced by the canonical system (4.1) and let M\ be the subspacc (4.3). It follows from (4.8) that 

(4.51) Nx = ker (T^a. - A) = {y G C\{I) : {y, Ay} e T^aJ, A G C. 

Assume also that M\ is a subspacc in %nax given by Nx = {{y, Ay} : y G A/a}, A G C. 

Definition 4.15. [19] The numbers Nj^ = dimA/i and = dimAA-j are called the formal deficiency 
indices of the system (4.1). 

It is clear that N± < n. Moreover, if the system (4.1) is regular, then Nj^ = N- = n. 
Next assume that 

^x:=^x {Tmin) = ker {Tmax - A) , A e C 

is the defect subspace and 

n± := n±{Tmin) = dimO^A, A G C± 

are deficiency indices of the symmetric relation Tmin in L'^{I). It is easily seen that ttA/a = O^Ia and 
ker (tt \ A/a) = A/" for each A G C. This implies the following proposition. 

Proposition 4.16. [19, 22] Given a canonical system (4.1). Then N± = dimMx, A € C± (i.e., dimTVA 
does not depend on A in either C+ or C-) and 

(4.52) N+=n+ + k^f, N_=n-+kM. 

As is known (see for instance [4]), for any closed symmetric relation A in ^3 the Neumann formula is 
valid. In the case of the minimal relation Tmin in T\{I) this formula is 

(4.53) + 0^a(T™„) + %(T™„), A e C \ M. 

In the following proposition we show that similar formulas hold for Tmin and Tmax- 

Proposition 4.17. Let Tmin o-nd Tmax be minimal and maximal relations in jC\{I) induced by the 
system (4.1). Assume also that M is the null manifold (4.4), A/"' C A/" is the subspace (4.46) and let 
kj\f' = dim A/"'. Then: 1) for each A G C \ R the following Neumann formulas hold 

(4.54) Tmax = Tm^n + {A + A^a)' ^min H (A^A + AA^) = A/"' © A/" = {{y, /} : y G AA', / G AA}. 
2) the following equality is valid 

(4.55) dim{Tmax/Tmin) = dim(dom7;„aa,/dom7;rji„) = N+ + N_ - kj,/ - k^>. 

Proof. Since TiTmax = Tmax, T^Tmin = Tmin and ttAa = Vlx{Tmin), it foUows from (4.53) that 

(4.56) Tmax = Tmin + {A + A^a) + (^ [ Tmax), A G C \ R 
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(clearly A/a (^J^x = {0}, so that the sum J\f\ +Mj in (4.56) is direct). Next, the inclusion y & J\f implies 
that y €J\f\n J\fj. Moreover, 

{^'0} = {-Ay^-^Ay} + {j^y'^i^^y} ■ 

and consequently 

(4.57) {y,0}eAfx+J^x (?/ e AA, AeC\M). 

Furthermore for each / e C\{1) such that A{t)f{t) = a.e on I one has {0, /} € l^in- This and (4.47) 
give the inclusion ker (tt |" Tmax) C Tmin + {■N'x +.^)> which together with (4.56) yields the first equality 
in (4.54). 

Let us prove the second relation in (4.54). If {y, /} € 7^m n (A/a +A/'^) , then 7r{y, /} e Tmi„n(?lA+OTx) 
and hence TT{y, /} = 0. Therefore by (4.48) y e J\f' and in view of (4.57) {y, 0} e A/a + A^x- Moreover, 
since {y,f} <S Aa + Ay as well, one has {0, /} G A/a + Nj- This implies that there exist y € Afx and 
z e A/^- such that y + z = and Ay + Az = /. Hence / = (A — A)2; = (A — X)y, so that / G A/a n A/^- and 
by (4.7) fGAf. Thus {y, /} e AA' AA. 

Conversely, let {y, /} € TV' ® A/" with y G Af' and / € TV. Then according to (4.46) {y, 0} S Tmtn 
and (4.57) gives {y, 0} G tVa + Afj. Therefore the inclusion {y, 0} G 7^i„ n (TVa + TV^) is valid. Next, 
{0, /} G T^in and the representation 

{0,/} = izx({/,A/}-{/,A/}) 

together with (4.7) shows that {0, /} G TVa + Afj. Thus {0, /} G Tmin n (TVa + Afj) and therefore 
{Uj /} € T^ira n (TVa +TVx) as well. This proves the second relation in (4.54). 

To prove (4.55) we first note that the equality r := dimlTmax /Tmin) = N++N-~kf^~k_\fi is immediate 
from (4.54). Next assume that {{yj, fj}}i is a basis of Tmax modulo Tmin- Then the immediate checking 
shows that {yj}i is the basis of domTmax modulo dom7^i„. Therefore dim(dom7^aj,/dom7^i„) = r(= 
dim{Tmax /Tmin)) which Completely proves (4.55). □ 

In the following proposition we give a somewhat different form of the Neumann formulas, which hold 

in the case of the regular endpoint. 

Proposition 4.18. Let Ta be the linear relation (4.31) induced by the system (4.1) with the regular 
endpoint a. Then 

(4.58) Tmax = Ta + {J<fx+J<fx), Ta H (A^A + AA^) = {0} 8 AA = {{0, /} : / G AA}, AgC\M, 
and the following equality holds 

(4.59) dim{Tmax/Ta) = dim(dom7^aa,/dom7^) = N+ + N_ - kj^. 

Proof Let TV' C TV be the subspace (4.46). Then by (4.57) TV" C TVa +Afx and the first equality in (4.54) 
together with (4.49) gives the first equality in (4.58). 

Next assume that {y, /} G 7^ fl (tVa + TV^). Since Ta C Tmin, it follows from (4.54) that y & Af' and 
f G J\f. Moreover, by (4.31) y{a) = and therefore y = 0. Hence {y, /} G {0}®TV. Conversely, in view of 
the second equality in (4.54) each pair {0, /} with / G TV belongs to tVa -fTV^ and obviously {0, /} G Ta- 
Hence {0,/} G 7^ n (tVa +Nj), which yields the second equality in (4.58). Finally, one proves formula 

(4.59) in the same way as (4.55). □ 

Proposition 4.19. Assume that the canonical system (4.1) has the regular endpoint a. Moreover, let 71 
be the linear relation (4.27) and let Hi = {y(a) : y G domTi}. Then Tmin C 71 C Tmax o,nd 

(4.60) Hi = (JH„)-L, dim(domri/dom7;) = n-k^f, 
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where Ma and kj^ are defined by (4.5) and (4.6) respectively. 

Proof. It follows from (4.45) that {Jy{a), z{aj) = for any y G domTi and z £ Af. Therefore Hi C 
(JHa)-"- and to prove the first equality in (4.60) it remains to show that (JHa)-"- c Hi. 
First assume that the system (4.1) is regular and let 

(4.61) H'l = {y{a) : y e domTmax and y{b) = 0}. 

Moreover, let A/'q be the subspacc in A/'o(= kerTmax) given by Afo ^ {y € Afo ■■ y{a) e H^}. Then 
Aq (lAf = {0} and, therefore, the equality {y,y)A = (y S AAq) implies y = 0. Hence Ag is a finite- 
dimensional Hilbert space with the inner product {y,z)A- 

Let h e (JH„)-L, so that Jh € H^. Then ip{z) = -{Jh,z{a)), z G J\f^ is an antilinear functional on 
A/'q and hence there exists //^ G A/'q such that 

(4.62) {fh,z)A^~iJh,z{a)), z ^ 

Next assume that y € ACiX) is the solution of the equation Jy' — B{t)y = A{t)fh{t) such that y{b) = 0. 
Then {y, /h} G Tmax and, consequently, t/(a) S H'^. Therefore y{a) G (JHo)-"-, which gives the inclusion 
Jy{a) e H^. Applying now the Lagrange's identity (4.15) to {y,fh} and {z,Q} {z e A/'q) and taking 

(4.62) into account one obtains 

-{Jh, z{a)) ^ ifh, z)a = -{Jy{a), z{a)), z € Af^. 

In this equality Jh e H^ , Jy{a) G M-^ and z{a) takes on any values from H^, when z run through A/'q. 
Therefore y{a) = h and, consequently, h € H'^. Thus (JHa)-"- c H'^, which together with the obvious 
inclusion H'l c Hi(c (JH„)-L) gives 

(4.63) Hi =H'i = (JH„)-L. 

Assume now that the system (4.1) is singular. For each finite segment I' = [a, /3] C I denote by Af^' 
the linear space (4.24) and let Hj = {y{a) : y e Af^ }. It is easily seen that H^ = Q^T'ci^a • Moreover, 
it was shown in the proof of Lemma 4.5 that there is a segment Xg = [a, /3o] such that C HJ' for all 
T C and H? = Hf for all I' D Tq. This implies that = H^. 

Next assume that Tmax is the maximal relation in C\{Iq) induced by the restriction of the system (4.1) 
onto I'q. Since this restriction is regular, it follows from (4.63) and (4.61) that for any h G (JHo)-'-(= 
(JHa")^) there exists {y, /} G Tnax such that y{0) = h and y(/3o) = 0. Continuing the functions y 
and / by onto I we obtain the pair {y, /} S 7i with y(0) = h. This yields the required inclusion 
(JH„)^ C Hi. 

Let us prove the second equality in (4.60). It follows from the first equality in (4.60) that ri := 
dimHi = n — kj^. Let {yjYi be a system of functions yj S domTi such that {yj{^)Yi is a basis in 
Hi. Then the immediate checking shows that this system forms a basis of domTi modulo dom?^, which 
yields the desired equality. □ 

Definition 4.20. The canonical system (4.1) is called definite if the corresponding null manifold A/" = {0}. 

The following corollaries are implied by the above results on arbitrary (not necessarily definite) canon- 
ical systems. 

Corollary 4.21. If the system (4.1) is definite, then N± = n± and the following Neumann formula holds 

(4.64) Tmax = Tmin + Af\ + Afj, A € C \ R. 

Proof. The desired statements are immediate from Propositions 4.16 and 4.17. □ 
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Corollary 4.22. Let the canonical system (4.1) with the regular endpoint a be definite. Then Ta = Tmin 
and for every /i S H there exists {y, /} G Tmax such that y(a) = h. If in addition the system is regular 
(that is, I = [a, \)\), then for any hi, /12 G H there is {y, /} € Tmax such that y{a) = hi and y{b) = h2- 

Proof. The first statement follows from Propositions 4.13 and 4.19. 

Next assume that the system is definite and regular and let fti, /i2 € H. Then by (4.61) and (4.63) there 
is {j/i, /i} e Tmax with 2/1 (a) = hi and yi{h) = 0. Moreover, by symmetry there is {j/2, /2} S Tmax with 
2/2(0) = and y2{b) — /i2. Clearly, the sum {y, /} = {yi, fi} + {3/2, /2} has the required properties. □ 

Remark 4.23. For the definite system (4.1) Theorem 4.12 and Corollary 4.22 were proved in [27]; the 
Neumann formula (4.64) was obtained in [22]. 

The general (not necessarily definite) canonical system of an arbitrary order n was considered in 
[22], where the minimal relation Tmin was defined as closure of To (see (4.14)) and then the equality 
T^j„(= Tq) = Tmax was proved . Note in this connection that our definition (4.12), (4.13) of Tmin 
seems to be more natural and convenient for applications; in particular cases of differential operators and 
definite canonical systems such a representation of Tmin can be found, e.g., in [26, 2]. Observe also that 
our Proposition 4.19 improves similar result in [22, Proposition 2.12]. 

5. Boundary relations for canonical systems and boundary conditions 

5.1. Boundary bilinear forms. In this section we suppose that the canonical systems (4.1) is defined 
on the interval I = [a, h) with the regular endpoint a. 

As is known the signature operator in (4.1) is unitary equivalent to 

/ -Ih\ 

(5.1) J= i6Ijj \:H®H®H^H®H®H, 

/ 

where 5 G {—1, 1} and H, H are finite-dimensional Hilbert spaces. The numbers 5, dim if and dim if are 
unitary invariants of J, which are defined by the following relations: if we let 

(5.2) = dim ker (i J — /) and = dimker (i J + /), 
then 

(5.3) 5 = sign(i^_ — u^), dimif = minji^^, dimif = \v_ — 
Using this fact we assume without loss of generality that 

(5.4) M = H e H @ H 

and the signature operator J in (4.1) is given by (5.1). 

Next consider the boundary bilinear form [•, •];, on domT^aj^ defined by (4.10). Clearly, this form is 
skew-Hermitian (that is [y, z]b = — [z.j/]^) and its kernel coincides with domTi, where 7i is the linear rela- 
tion (4.27). Moreover, since dom7^i„ C domTi C domT^oa; and by (4.55) dim(dom7^a2;/dom7^i„) < 
00, there exists a (not unique) direct decomposition 

(5.5) dom Tmax = dom Ti + X»6+ + T^b- 
such that 

(5.6) 1^6+ := diml?b+ < 00, u^- := dimI3b_ < 00 
and the following relations are valid 

(5.7) lm[y,y];,>0, 0=^yeVh+; Im[z,z]6<0, O^zeVb-; [y,z]b^O, y€Vb+,zeVh^. 

As is known the numbers and Vb- are called indices if inertia of the form [•,•];,. These numbers are 
uniquely defined by the form and do not depend on the choice of the decomposition (5.5). 
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Lemma 5.1. Let [•, be the bilinear form (4.10) with the indices of inertia (5.6) and let 6b := sign(i/(,+ — 
) . Then: 1 ) there exists Hilbert spaces Ht and Hb and a surjective linear map 

(5.8) Tb = (Fob : f ^ : TuV ■ domTmax ^Ub^Ub^nb 
such that 

(5.9) [y, z\b = iSbitby, tbz) - {TibV, ^obz) + (roty, Tibz), y,zG domTmax- 
Letting H;, :='Hb®'Hb®^b o-nd introducing the signature operator Jb € [H;,] by 

( -/„A 

(5.10) Jfc = idblf^^ \ ■.Hb®Hb®Hb^Hb®Hb®nb 

J 

one can represent the identity (5.9) as 

(5.11) [y,z]b = {Jbrby,Tbz)Mt, y,z GdomTmax- 

2) if a surjective linear map Tb of the form (5.8) satisfies (5.9) , then kerF^ = domTi and 

(5.12) dim Kb = minlub+jPb-}, dirnHb = \i'b+ — i^b-\- 

Proof. 1) Assume for definiteness that i'b+ > I'b-, so that 5b = 1. It foUows from (5.7) that 'Db+ and Vb- 
are finite-dimensional Hilbert spaces with the inner produets (yi,j/2)+ = —i[yi,y2]b, 2/1)2/2 S ©6+ and 
{zi,Z2)- = i[zi,Z2]b, zi,Z2 S 'Db- respectively. Moreover, by (5.5) 

(5.13) dom7;„„^ = domTi + {Hb ® Hb) + 'Db-, 

where Hb and Hb are subspaces in such that dim'Hb = i'b-{= dim !)(,_) and 'Db+ = Hb ® Hb- 
Let y be a unitary operator from onto Hb and let 

(5.14) h = Vn,, rob = j^{Vn, + VVv,_), = -^{Vh, -VVv,_\ 

where V^^, Vm and are the skew projections onto the subspaces Tib, Hb and Vb- corresponding 

to the decomposition (5.13). The immediate checking shows that the map Tb given by (5.8) and (5.14) 
is surjective and satisfies (5.9). 

Similarly one proves the statement 1) in the case i'b+ < i^b- ■ 

The statement 2) immediately follows from surjectivity of Tb and the identity (5.9). □ 

Remark 5.2. One can show that the map Tb in Lemma 5.1 can be represented in the more explicit form. 
Namely, it is not diflficult to prove that there exist systems of functions {^/'jli', and {Oj}'{*' in 

domT^aj, with Ub = min{i/(,+, Vb-} and />(, = \ub+ — ^^6-1 such that the operators 

Tofcy = {[y,V'j]b}iN ^bV = {[g,^Pj]bYi, Tuy = {[y,ej\bYi , y & domTmax 

form the surjective linear map Tb = {Tob '■ ^b '■ Tib)^ '■ dom7^„,aa; — ^ C^' ® © C^*" satisfying the 
identity (5.9). This assertion shows that for each y e domTmao: the elements T^by^Tiby and Tby are, in 
fact, boundary values of the function y{-) at the endpoint b. 
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5.2. Decomposing boundary relations. Assume without loss of generality that the Hilbert space H 
and the signature operator J in (4.1) are defined by (5.4) and (5.1) respectively. In this case each function 
y{-) e domTmax admits the representation 

(5.15) y(t) = {yo(t),y(i), J/iWKeH), tel, 

where yo{t),y{t) and yi{t) are components of y{t) corresponding to the decomposition (5.4). 

Let v+ and by given by (5.2) and let ui,+ and be indices of inertia (5.6). Then according 
to Lemma 5.1 there exist Hilbert spaces T-Li, and T-Lb satisfying (5.12) and the surjective linear map 
Ff, = (Fo;, : F;, : Fi;,)^ such that (5.9) holds. Without loss of generality assume that 

(5.16) - Vb- >i'--y+ 

and consider the following three alternative cases: 
(i) v_-v+>Q 

It follows from (5.3) and (5.12) that in this case 

(5.17) dimi? = i/_|_, 6.\m.H = v_ — u^, dim'Hb = f(,_, dirnHb = i^b+ — i^b- 

and the inequality (5.16) gives dimff < dim'Hf,- Therefore without loss of generality we can assume that 
ff is a subspace in Hb- Letting ^2 = 'Hb Q H, we obtain jib ='H2(B H, so that the operator Fj, in (5.8) 
admits the block representation Fb = (r26 : Fib)^ : domTmax "^2 ® H- Put 

Ui=H®H®'Hb, Uo = U2®Ui='H2®H®H®Ub 
and introduce the operators 



(5.18) T'^y = {V2by, yo{a), -^{y{a) - f 152/)}, Toby}{€ H2®H®H® Hb), 

(5.19) T[y = {yi{a), -^{y{a) + tibv), -^iby}{& H®H® Hb), y e domT;^^^. 

(ii) u- — u+ <0 and f6+ — Ub- > 0, so that 

(5.20) dimH = U-, dim = zv+ — z/_ , dimHb = i'b-, dim Hb = i'b+ — t^b-- 
In this case we put 

H2 = H®Hb, Hi = H®-Hb, Ho = 'H2®ni = {H®Hb)®H®'Hb 

(5.21) T'ov = {{y{a), hy}, yo{a), Toby}{e {H®'Hb)®H® Hb), 

(5.22) F'ly = {j/i(a), -Tuy}{e H ® Hb), y e domTmax- 

(iii) Ub+ — Ub- < 0, so that 

(5.23) dimi7 = i^_, dim = j/+ — , dimH;, = z^b+i dimHb = Vb- — Vb+ ■ 



In view of (5.16) one has dim?^b < dim/f, which enables us to assume by analogy with the case (i) 
that Hb C H. Letting H2 = H Q Hb one obtains H = H2 ® Hb, which implies the representation 
y{t) = {mit), yb{t)}i^ H2®Hb) of the functions y{t) from (5.15). 
In the case (iii) we let 

Hi=H®Hb®Hb, Hq = H2®Hi =H2® H ®Hb®Hb 

(5.24) F'oy = {y2(a), ?yo(a), -^iyb{a) -fby)},Toby}{eH2® H ®Hb®Hb) 

(5.25) r[y = {yi{a),^{yb{a) + rby),-riby}{€H®Hb®Hb), y € domTmax- 

Note that in each of the cases (i) (iii) Hi is a subspace in Hq, H2 = Hq Hi and F^ is a linear map 
from domT^aa; to Hj, j G {0.1}. Moreover, formulas (5.17), (5.20) and (5.23) imply that in all cases 
(i)-(iii) 
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(5.26) dim "Ho = v+ + Vb+, dim "Hi = v- + Vb-. 

Theorem 5.3. Assume that a is a regular endpoint for the canonical system (4.1) and the inequality 
(5.16) is satisfied. Moreover, let Hj be Hilbert spaces and T'j : domTmax "^j, j € {0, 1} be linear maps 
constructed for the alternative cases (i)-(iii) just before the theorem. Then the equality 



defines the boundary relation T : {L\{I)Y — > "Hq ® "Hi for T^axi— T^min) with 

(5.28) dim-Ho=iV+ and dim-Hi==iV_. 

Proof. Let us show that the hnear relation (5.27) satisfies the assumptions of Corollary 3.7 for A Tmin- 
Assume that V = (FJ, : T\y : dom7;,jax Hq®^!. Then definitions (5.18) (5.25) of and T'^ and 

the equality kerFf, = domTi (see Lemma 5.1, 2)) imply that kerF' = 7^. Therefore by (5.27) one has 

kerF = TrkerF' = ttT^ = T^in- Moreover, it follows from (5.27) that domF = nTmax = T^ax- 

Next, the immediate calculations with taking (5.9) into account show that in each of the cases (i)-(iii) 

the operators Fq and F'^ satisfy the relation 

[y,z]b-{Jy{a),z{a)) = {r[y,T'oz) - {T'oy,T[z) +i{P2T'oy,P2T'oz), y,ze domTmax- 

This and the Lagrange's identity (4.11) give the identity (3.6) for F. 
Now it remains to prove (3.41). It follows from (5.5) and (5.6) that 

i'b+ + Vb- = dim(dom7^aa;/dom7i) = dim(dom7^a2:/dom7^) — dim(dom7i/dom7^). 

Combining this equality with (4.59) and the second equality in (4.60) one obtains 

Vb+ + n- = {N+ -\-N_- kM) - (n - k^) = N+ + N_ - n. 

This and (5.26) give 

(5.29) dim(-Ho ® "Hi) = {v+ + v_) + {vb+ + Vb-) = n + {N+ + N_ - n) = N+ + N_. 

Next, in view of (5.27) one has mulF = {F'y : {y,f} & ker(7f |~ Tmax) for some / e £^(I)} and 
(4.47) yields 

(5.30) mulF = T'N = {{T'^y.T'^y) -.y&M}. 
Since obviously ker (F' \ M) = {0}, it follows from (5.30) that 

(5.31) nr(=dim(mulF)) =dimA/' = A;Ar. 
This and (4.52) imply that 

(5.32) n+ + n_ + 2nr = {N+ - k^) + {N^ - k^) + 2kj^ = N++N_. 
Combining now (5.29) and (5.32) wc arrive at the required equality 

dim(-Ho ® Til) = n+ + n_ + 2nr. 

Thus according to Corollary 3.7 formula (5.27) defines the boundary relation F for Tmax- Moreover, 
combining (3.40) with (5.31) and (4.52) we obtain the equalities (5.28). □ 

Definition 5.4. The boundary relation F : (L^(I))^ Ho (BHi constructed in Theorem 5.3 will be 
called a decomposing boundary relation for T^ax- 
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Proposition 5.5. The formal deficiency indices of the canonical system (4.1) with the regular endpoint 
a can be calculated via 

(5.33) N+ = iy+ + f 6+ , iV_ = !/_ + u^- , 

where z/± are the numbers (5.2) and Vb^, v^^ are indices of inertia of the bilinear form [■,-\b- It follows 
from (5.33) that in the case of the regular endpoint a the following inequalities hold 

(5.34) v+<N+< n, v- <N^ < n. 

Proof. If i/± and i'b± satisfy (5.16), then the equalities (5.33) follow from (5.28) and (5.26). In the 
opposite case Vf,^ — v^- < i'- — 1^+ the equahties (5.33) can be obtained by passage to the system 

-Jy'{t) + B{t)y{t)=mf{t)- 

□ 

In the case = N- the construction of the decomposing boundary relation for T^ax can be rather 

simphfied. Namely, the following corollary is valid. 

Corollary 5.6. Assume that a is a regular endpoint for the canonical system (4.1). Then: 
1 ) this system has equal deficiency indices iV+ = A''_ if and only if 

(5.35) Vb+ - n- =y- -vjr 

2) if = then there exist a Hilbert space Hb with dimHb = min{ub+, Ub-} o,nd a surjective linear 

map 

(5.36) = {Fob : f ^ : Fu)'^ : domT„,ax Hi, ® H @ 

such that the identity (5.9) holds with 6b = 6{= sign(z^_ — Moreover, for each such a map Fi, the 

equality 

'{yo{a), -^5(y(a) -Fby), Fobv}' 
v{yi(a), -^{yia) + Fby), -Fuyjy 



(5.37) 





defines the decomposing boundary relation F : (L^(I))^ — ^ {H (B H 
In the case of the regular system (4.1) one can put Hb = H and 

[ [ VfJ \{yi{a), ^{y{a) + y{b)), -yi{b)}J J J 

Proof. The statement 1) follows from (5.33). 

2) Combining (5.35) with (5.3) and (5.12) one obtains dim'Hb = dim J?. Therefore one can put in (5.8) 
Hb = H, in which case the map takes the form (5.36) and the equality (5.27) for F can be represented 
as (5.37). In the case of the regular system one can put Hb — H and Fby = {yo{b), y{b), yi{b)}{& H), so 
that the equality (5.37) takes the form (5.38). □ 

Corollary 5.7. Assume that the canonical system (4.1) with the regular endpoint a has minimal formal 
deficiency indices = and = . // > , then the equality 

(5.39) r = 1 : {y,f} e r_| 



defines the decomposing boundary relation T : {L\{I)Y — >■ {H (B H)(B H for Tmax- 

Proof. Since 1/^+ = Vb- = 0, it follows from (5.12) that Hb = Hb = {0}. Combining this equalities with 
(5.21), (5.22) and (5.27) we obtain the representation (5.39) for F. □ 
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5.3. Boundary conditions for definite systems. As is known (see for instance [22])the maximal 
operator Tmax induced by the definite system (4.1) possesses the following property: for each {y, f} G 

Trnax thcrc cxlsts a uniqiic fimction y G AC {I) such that y (z y and {y, /} G Tmax for each / G /. Bellow, 
without any additional comments, wc associate such a function y G AC{X) with each pair {y, /} G Tmax- 

Theorem 5.8. Let under the conditions of Theorem 5.3 the canonical system (4.1) he definite. Then: 

1) The operators Tj : T^ax ^-j, j € {0, 1} given by 

(5.40) To{yJ} = T'oy, T,{yJ} = T[y, {y,7}eT„„, 

form the boundary triplet H = {Ho ® Hi, Fq, Fi} for T^ax- 

In the case of minimal deficiency indices n+ = and n_ = U- one has Ho = H ® H, Hi = H and 
the equality (5.40) takes the form 

(5.41) To{yJ} = {y{a),yo{a)}{^II®H). Ti{yJ}^yi{a){^H), {yJj&Tmax- 

2) If n+ = n-, then the statement 2) of Corollary 5.6 holds and the decomposing boundary relation 
(5.37) turns into the boundary triplet IT = {H, Fq, Fi} for Tj^ax with H = H ®H ®Hb and the operators 
Tj : Tmax ^ given by 

(5.42) To{y, /} = {t/o(a), ^S{y{a) - hy), Fo6t/}(e H®H® Hb), 

(5.43) Ti{yJ} = {yi{a), ^{y{a)+tt,y),-Tit,y}{€H®H®Hb), {yJ}eTmax, 

with 5 = sign (i/_ — z^+). In the case of the regular system (4.1) one can put H = H ® H ® H and 

(5.44) Fo{y, /} = {yo{a), -^5{y{a) - y{b)), yo(b)}(e H ® H ® H), 

(5.45) Vi{yJ} = {yi{a), ^{y{a) + y{b)), -yi{b)}{& H ® H ® H), {yJ}eTmax- 

Proof. 1) Let F be the decomposing boundary relation (5.27) for Tmax- Then by (5.30) mulF = {0} and 
Corollary 3.6, 2) implies that the operators (5.40) form the boundary triplet 11 = {Ho ® ■Hi,Fo,Fi} for 
Tmax- Moreover, in the case n± = v± the equality (5.39) gives (5.41). 

The statement 2) of the theorem follows from Corollary 5.6, 2). □ 

In the sequel the boundary triplet 11 = {Ho ® "Hi, Fg, Fi} defined in Theorem 5.8, 1) will be called the 
decomposing boundary triplet for Tmax- In the case of equal deficiency indices n+ = n_ such a triplet 
takes the form E = {H, Fq, Fi}, where H = H ® H ®Hb a.nA Fq, Fi are defined by (5.42) and (5.43). 

Proposition 5.9. Let the minimal relation Tmin induced by the definite system (4.1) with the regular 
endpoint a has equal deficiency indices — n_ and let 11 = {?^,Fo,Fi} be the decomposing boundary 
triplet (5.42), (5.43) for Tmax- Then for each operator pair (linear relation) 9 = {(Co,Ci);/C} given by 
the block representations 

(5.46) Co = {Coa--Ca:Cob):H®H®Hb^lC, Ci = {Cia ■ Cb : Cu) ■- H ® H ®Hb ^ K 
the equality (the boundary conditions) 

(5.47) A = {{y, /} e Tmax : CoaVoia) + Cay{a) + CiaViia) + CobTobV + Cbhy + CibTuy = 0} 

defines a proper extension A of Tmin o,nd, conversely, for each such an extension there is a unique 

admissible operator pair (linear relation) 9 = {(Co,Ci);/C} given hy (5.46) and such that (5.47) holds. 
Moreover, the extension (5.47) is maximal dissipative, maximal accumulative or self-adjoint if and only 
if the operator pair (linear relation) 9 = {{Co, Ci)',f^} with 

(5.48) Co = {Coa --^{Ca- Cb) : Cob), Ci = (Ci„ : (^ + 4) : -Cu) 
is maximal dissipative, maximal accumulative or self-adjoint respectively. 
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Proof. It follows from (5.42) and (5.48) that the boundary conditions (5.47) can be written as 

(5.49) A = {{y, /} G T^a. : C^^^iv. /} + C^V^{y, /} = 0}. 

This and Proposition 2.6 yield the desired statements. □ 
In the following corollary we give a somewhat different description of proper extensions A G Extr^^^ ■ 

Corollary 5.10. Assume that a is a regular endpoint for the definite canonical system (4.1) and n+ = 

n_ TO. Let Hb be a Hilbert space, let Jb G [Hb] be a signature operator and let Tb : domT^ax — > Hf, 
be a surjective linear map such that (5.11) holds (according to Lemma 5.1 such Mb, Jb and Tb exist and 
dimMfe = Vb+ + i'b+)- Moreover, letK, he a Hilbert space withdimJC = m, letCa € [H,/C] andCb G [Hb,/C] 
be operators such that ran (Co : C^) = /C and let A G Extr^..^ he an extension given by 

(5.50) A = {{y, /} G Trr^a. : ^^(a) + CbTby = 0}. 

Then A is maximal dissipative, maximal accumulative or self-adjoint if and only if 

(5.51) i{CaJC:-CbJbC*b)<0, i{C^JCl-CbJbCl)>Q or C^JCl = CbJbCt 
respectively. 

Proof. Assume without loss of generality that H;, = T-Lb (B H (B T-Lb and the operators Jb and Tb are of 
the form (5.10) and (5.8) respectively (with H in place of Hb). Then according to Theorem 5.8 the 
Hilbert space H = H (B H (BKb and the operators (5.42), (5.43) form the (decomposing) boundary triplet 
n = {H, To, Fi} for Tmax- Next assimae that 

Ca = {Coa ■■ Ca : Cia) : H ® H ® H ^ K, a = {Cob : a : Cib) :nb®H®nb^lC 

are the block representations of Ca and Cb and let Co and Ci be given by (5.48). Then (5.50) can be 
written as (5.47) and according to Proposition 5.9 A is maximal dissipative, maximal accumulative or 
self-adjoint if and only if the operator pair 6 == {(Co, Ci); /C} belongs to the same class. The immediate 
calculations show that 

2Im(CiCo*) = i{CbJbC; - CaJCl). 

Moreover, since ran [Ca '■ C),) = /C, it follows that the operator pair (Cq : Ci) is admissible. Applying 
now Proposition 2.3, 2) we arrive at the required statement. □ 

Definition 5.11. The boundary conditions (5.47) are said to be separated if there exists a decomposition 
)C = tCa ® fCb such that the operators (5.46) are 

(5.52) Co=ft" :iJ©iJ©m^/Ca®/C6 



Noa 


















Nb 



(5.53) Ci = 'J" :H®H®nb^]Ca®]Cb 
and, consequently, the equality (5.47) takes the form 

(5.54) A = {{y, /} G T^ax ■ NoaVoia) + Nay{a) + NiaViia) = 0, NobTobV + mtby + NuTuy = 0}. 

The separated boundary conditions (5.54) will by called maximal dissipative, maximal accumulative 
or self-adjoint if they define the extension A of the corresponding class. 

With the separated boundary conditions (5.54) we associate the operators 

(5.55) Sa = Im{N,aN*J + iN^N* , Sb = Im{N,bN*t,) + ^MbN; , 

(5.56) Na = {Noa - iNia ■■ -iV2Na) : H ® H ^ K^, Nb = {iV2Nb : Nob - iNu) : H ^Ub ^ ICb- 
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Theorem 5.12. Let for simplicity v- > i>+ and let the assumptions of Proposition 5.9 be satisfied. Then: 
1) the separated boundary conditions defined by (5.52) -(5.54) are maximal dissipative if and only if 

(5.57) Sa>0, Sb<0 and e p{Na) n p{Nob + iNu) , 
in which case the following equalities hold 

(5.58) dim/Ca = i^-, dim /C;, = f6_ . 
The same boundary conditions are maximal accumulative if and only if 

(5.59) Sa<0, Sb>0 and e p{Noa + iNia) n p{Nb), 
in which case 

(5.60) dim /Co = dim/C^ = L'b+ 

(here ICa o,nd tCh are Hilbert spaces from (5.52) and (5.53) ). 

2) self-adjoint separated boundary conditions exist if and only if v- = or, equivalently, if and only 
ifW = H ® H and the operator J in (4.1) is 

J = [J^ : H ® H ^ H ® H. 

If this condition is satisfied, then: 

(i) the decomposing boundary triplet (5.42), (5.43) takes the form 11 = {'H,ro,ri}, where % = H(B'Hb 
and the operators Fj, j e {0, 1} are given by 

(5.61) To{yJ} = {yo{a),Toby}{€H®nb), Ti{yJ} = {yi{a),-Tiby}{€ H ®nb), {yJ}&Tmax 

(ii) the general form of self- adjoint separated boundary conditions is 

(5.62) A = {{y, J] e Tmax : A^Oayo(a) + A^ia2/i(a) = 0, NobToby + NuTuy = 0}, 

where the operators Nja € [H,)Ca\ (^nd Njb € [HbjlCb], j € {Oil} (f^e components of self-adjoint operator 
pairs Oa = {(A^oa, ^la); /Ca} and Ob = {{Nob,Nib);}Cb}- 

Proof. 1) Let Co and Ci be the operators (5.48) corresponding to the separated boundary conditions 
(5.54). Then in view of (5.52) and (5.53) one has 

(5.63) Co={ " f^^ ^. \:H®H®Hb^K,®Kb 

V ^ Nf,b) 

(5.64) '^i=(^0 %^b 

Combining now the last statement in Proposition 5.9 with formulas (2.3) and (2.4) we obtain the following 
assertion: 

(a) the boundary conditions (5.54) are maximal dissipative (resp. maximal accumulative) if and only 
if l^iCiCl) > and e p{Cq - iCi) (resp. Im(C'iC'c;) < and G p{Co +iCi)). 
It follows from (5.63) and (5.64) that 



' ° V h^hK -N^bN*o,-iNbN*b) 



2 ■""■"a -"iO-^'Ob 2 

and, consequently, Im(CiCo) = diag(6'a, —Sb)- Hence the following equivalences are valid 

(5.65) Im(Ci5o) > <^ 5a > and Sft < 0; Im(CiCo) < <^=^ S'o < and > 0. 
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Moreover, by (5.63) and (5.64) one has 

\ u iVo6 + tiyibj 

which yields the equivalences 

(5.66) G p{Co - iCi) ^ e piNa) n p{Nob + iNu), e p(Co + iCi) ^ G p(Afoa + iNia) n ^(iVfc). 

Now assertion (a) together with (5.65) and (5.66) gives the required description of all maximal dissipative 
and accumulative separated boundary conditions by means of (5.57) and (5.59). Moreover, (5.57) implies 
that 

dim/Ca = dim(if ® H), dim/C^ = dim'Hb, 

which in view of (5.3) and (5.12) leads to (5.58). Similarly one proves the equalities (5.60). 

2) Since self-adjoint separated boundary conditions are simultaneously maximal dissipative and maxi- 
mal accumulative, it follows from (5.58) and (5.60) that an existence of such conditions yields the equality 
= v+. Moreover, if this equality is satisfied, then the general form (5.62) of self-adjoint separated 
boundary conditions follows from the statement 1) of the theorem. □ 

Remark 5.13. 1) Theorem 5.12 enables one to introduce the important class of maximal accumulative 
(dissipative) separated boundary conditions, which consist of the self-adjoint condition at the regular 
endpoint a and the maximal accumulative (dissipative) condition at the point h. If, for instance, > 
then such separated conditions are defined by 

A = {{y, 7} € T^ax ■■ NoaVoia) + NiaViia) = 0, NobVobV + NtTby + NuTuy = 0}, 

where the operators Noa and Nob form the self- adjoint pair 9a = {iNoa,Nia);ICa}, while the operators 
Nob, Nib and Nb form the maximal accumulative pair 9b = {(("^-^6 • -^ofe), ("^f ' ~Nib))',l^b}- 

2) For a regular definite system (4.1) one can put in Corollary 5.10 Hf, = H, Jb = J and TbU = y{b),y G 
domT^aa;, in which case this corollary gives the following well known statement [12, 27]: the extension 
A = {{y, 7} G Tmax ■■ Cay (a) + Cby{b) = 0} is self-adjoint if and only if CaJC* = CbJC;. The case of 
the singular endpoint 6 under the additional assumptions vj^ = v- and myAT^ax = {0} was considered 
in the paper [20], where the criterium for sclf-adjointncss of the boundary condition (5.50) in the form of 
the last equality in (5.51) was obtained. Note in this connection that our approach based on the concept 
of a decomposing boundary triplet seems to be more convenient. In particular, such an approach made 
it possible to describe in Theorem 5.12 various classes of separated boundary conditions. 
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